Telemetry

The executor measures itself through the OpenTelemetry API and ships no exporter of its own. Where no SDK has been registered the API hands out no-op implementations, so an unmeasured process pays for a few function calls and publishes nothing at all. Registering the providers belongs to whoever owns the process: the daemon in one deployment, the host application in another. The operator guide is where a deployment turns it on.

More than one surface carries the same facts. Metrics are for a scrape, spans are for a trace, and the pool also publishes its own typed events. Library hosts read the numbers off those events without putting an SDK in the process at all. Everything below carries the instrumentation scope @supolka/codemode-executor, declared in packages/executor/src/spans.ts. That scope also becomes the otel_scope_name label on every scraped series.

No lane of pnpm verify reads this page against the instruments, so the tables here are a snapshot. But the scrape further down is not: it came out of a running server through the exporter the daemon configures.

Metrics

The pool builds its instruments in packages/executor/src/pool/instrumentation.ts and the server builds its own in packages/executor/src/server/instrumentation.ts. Each row’s instrument column names the OpenTelemetry instrument kind it was created with. A dashboard is built against the attributes column: it names every dimension a series splits on. No other dimension ever reaches these instruments.

Name	Instrument	Unit	Attributes	What it records
executor.execution.duration	Histogram	ms	outcome is success or error	The wall time of one execution, from the moment the worker is leased to the moment it settles.
executor.execution.wait	Histogram	ms	none	How long an execution waited for a worker before it started.
executor.tool.duration	Histogram	ms	none	The round trip of one tool call, from the sandbox to the caller’s dispatch and back.
executor.execution.running	UpDownCounter	none	none	Executions running right now.
executor.execution.rejections	Counter	none	code is the ExecutorErrorCode of the fault	Executions turned away without running, split by the code the caller met.
executor.worker.spawns	Counter	none	none	Workers that answered their first heartbeat.
executor.worker.crashes	Counter	none	none	Worker deaths outside a shutdown.
executor.worker.retires	Counter	none	reason is executions or memory	Workers the pool replaced after a drain.
executor.handles.closes	Counter	none	reason is closed, expired, terminated, crash or shutdown	Retained executions whose handles ended.
executor.pool.queue.depth	Gauge	none	none	Executions waiting for a worker, recorded by every lease attempt.
executor.connection.active	UpDownCounter	none	none	Caller connections open on the server right now.
executor.connection.rejections	Counter	none	none	Connections the onConnection hook turned away.

The crash reason stays out of executor.worker.crashes on purpose, because it is free text and would give the series unbounded cardinality. Whoever wants the words finds them on the worker.crash event below.

executor.pool.queue.depth is recorded by every execution as it leases a worker, so a replica carries the series from its first execution and not from its first queued one. A fresh replica therefore reports a depth an autoscaler can read, and the operator guide builds one around exactly that.

Instruments are not built until the first event reaches them, and a scrape shows only what has been recorded. executor.connection.rejections is missing from a healthy server’s output entirely, but it appears the moment a connection is first turned away. So a dashboard panel that expects the series from boot reads as broken while the server is healthy. A panel written to tolerate the absence does not.

How a Prometheus scrape renames a metric

The exporter rewrites every name on the way out, so a dashboard queries something other than the names in the table above. @opentelemetry/exporter-prometheus applies the rules below, following the OpenTelemetry rules for Prometheus and OpenMetrics.

OpenTelemetry	Prometheus
A dot in the name	An underscore
A Counter	The same name with _total, typed counter
An UpDownCounter	The same name with no suffix, typed gauge
A Gauge	The same name with no suffix, typed gauge
A Histogram	Three families, _count, _sum and one _bucket series per boundary
An attribute	A label of the same name
The instrumentation scope	The label otel_scope_name on every series

So executor.worker.crashes is scraped as executor_worker_crashes_total, executor.pool.queue.depth as executor_pool_queue_depth, and executor.execution.duration as the executor_execution_duration_count, executor_execution_duration_sum and executor_execution_duration_bucket families. One thing does not travel into the name: the unit. Prometheus practice puts the unit on the end of a metric name, so a reader coming from that convention looks for a _milliseconds suffix and does not find one. Wherever the unit column above says anything other than none, this exporter writes a # UNIT line instead. That line belongs to OpenMetrics and not to the Prometheus text format, whose parser ignores any # line whose first token is neither HELP nor TYPE. Durations are milliseconds. Their scraped names do not say so.

Below is real output, trimmed. A one-worker server ran a single execution and answered GET /metrics with text/plain; version=0.0.4; charset=utf-8, the content type the Prometheus text format states. Trimming drops bulk and not shape: the other families of that scrape, and all but one bucket of the histogram, which really runs from le="0" to le="+Inf".

# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_name="codemode-executor",telemetry_sdk_language="nodejs",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="2.10.0"} 1
# HELP executor_connection_active Caller connections open right now
# TYPE executor_connection_active gauge
executor_connection_active{otel_scope_name="@supolka/codemode-executor"} 1
# HELP executor_execution_duration Sandbox execution wall time
# UNIT executor_execution_duration ms
# TYPE executor_execution_duration histogram
executor_execution_duration_count{outcome="success",otel_scope_name="@supolka/codemode-executor"} 1
executor_execution_duration_sum{outcome="success",otel_scope_name="@supolka/codemode-executor"} 4
executor_execution_duration_bucket{outcome="success",otel_scope_name="@supolka/codemode-executor",le="5"} 1
# HELP executor_worker_spawns_total Workers that answered their first heartbeat
# TYPE executor_worker_spawns_total counter
executor_worker_spawns_total{otel_scope_name="@supolka/codemode-executor"} 1

Each family’s # HELP text is the instrument’s own description, so a grep through a scrape finds an instrument by what it means and not by the name it was renamed to.

target_info is not an instrument of this executor. It is the resource the SDK was configured with, published as a series so a query can join against it, and service_name there is whatever the host called the process.

Events a pool publishes

WorkerPool exposes a typed EventEmitter as events, and the metrics above are one subscriber of it. Library hosts that want the facts without registering an SDK subscribe to the same emitter, and so does the daemon, to count what its status command reports. Events fire whether or not the metrics are on. Passing instrumented: false in PoolOptions builds no instruments and subscribes nobody, and the same option on ServerOptions does it for the connection metrics and the request span. Pools remove every listener on shutdown. Payload types are declared in packages/executor/src/pool/events.ts and exported from the ./pool subpath.

Event	Payload	When it fires
execution.start	{ executionId, workerId, wait }	A worker has been leased and the execution is about to run. wait is how many milliseconds passed before the lease.
execution.settle	{ executionId, workerId, duration, outcome }	The execution settled, whether it returned a result, reported an error or threw a fault.
execution.reject	{ code }	execute rejected before the execution ran, because the lease failed.
worker.spawn	{ workerId, pid }	A child answered its first heartbeat.
worker.crash	{ workerId, pid, reason, runningExecutionIds }	A child died outside a shutdown. The identifiers name the executions that died with it.
worker.retire	{ workerId, reason, executions, memory }	A drained worker was replaced, on its execution count or on its process memory.
handles.close	{ executionId, workerId, reason }	A retained execution ended, once per execution however many closers raced.
queue.depth	{ depth }	Every lease attempt, every enqueue, and every dequeue, whether it served a lease or emptied the queue on shutdown.

Two different identifiers travel under the name executionId and they are not interchangeable. One is the pool’s own counter, execution-1 and upward: it identifies a run inside one pool and appears in execution.start and execution.settle. The other is the granted executionId of a retained execution, a UUID the sandbox mints and handles.close carries. Callers hold that one.

Spans

Both spans end with the work they wrap. On a failure the span records the exception, takes the ERROR status with the failure’s message, and ends before the failure is rethrown. One code path settles both.

Span	What it wraps	Attributes
executor.request	One execute call arriving at the server, from the RPC entry to the answer	executor.identity, the identity the onConnection hook returned, or anonymous where no hook is configured
pool.execute	One execution’s call into its child, inside the pool	executionId, workerId

executor.request is started as the active span, so pool.execute becomes its child wherever the host registered a context manager. When a collector is configured, the daemon registers one with its tracer provider. Without a context manager both spans are still recorded, but pool.execute is a root of its own instead of a child. A tool call takes no span at all and is measured as executor.tool.duration instead, because a span per call would multiply the trace by the number of tools a program happens to use.

Turning telemetry on

A daemon reads its own environment. It registers a Prometheus reader so that /metrics answers a scrape with no configuration, sends traces and metrics through OTLP when a collector is named, and stops measuring entirely under OTEL_SDK_DISABLED. Each of those settings is stated in the operator guide. Hosts that embed the executor register their own meter and tracer providers through the OpenTelemetry API before the first execution, and the instruments bind on the first event they see. A host that serves the protocol itself and wants a scrape endpoint passes metrics to createExecutorServer: a MetricsSource carrying a content type and a render function. Beside /healthz and /readyz, the server answers GET and HEAD on /metrics with what that renders.