The server and the daemon

The pool ends at a function call. Past it, a service answers for four things on its own: who may connect, whose numbers a run is bounded by, how the process reports its health, and how it dies when a platform tells it to. packages/executor/src/server owns the connection, the bounds and the probes; apps/daemon owns the process. Both stay separable on purpose. Node hosts that already own their lifecycle embed the server through the ./server subpath of @supolka/codemode-executor and keep their own configuration, logging and signals. One composition of it is the daemon, with a command line, telemetry, an admin app and a stop handler around it.

The executor protocol states what crosses the wire, and the operator guide what a deployment decides. This chapter is the machinery under both, and it takes the pool as given.

Accepting a connection

createExecutorServer in packages/executor/src/server/listener.ts holds one http.Server, one ws server in noServer mode, and one WorkerPool built from the pool options it was handed. The HTTP server answers the probes. Every execution arrives through a WebSocket upgrade.

handleUpgrade reads the acceptance flag before anything else. Servers that have stopped accepting write 503 Service Unavailable on the socket and then destroy it. Sockets destroyed with nothing written on them reach the caller as a network fault, not as an answer. Acceptance is a switch. stopAccepting and resumeAccepting flip it, open connections and their running executions are untouched, and readiness reports the new state. But once close() has begun, resumeAccepting throws the INTERNAL fault instead.

Next comes the identity hook. onConnection receives a ConnectionHandshake carrying the request headers flattened into a string map, with repeated values joined by a comma and a space, and the remote address of the socket. It answers one of two shapes:

export type ConnectionDecision = { identity: string } | { reject: true; reason?: string }

Servers with no hook name every caller anonymous. Daemons started from the command line do exactly that, because the command line has no flag for a hook. Hooks that throw are read as a rejection and the failure is logged, so a policy that breaks refuses the connection instead of admitting it. A rejection writes 403 Forbidden and carries no reason: a reason composed by the caller’s own policy would land inside a response header and open the response to header injection. That reason goes to the logger beside the remote address, and the rejection counter rises. The hook may be asynchronous and may resolve after close() began. So the acceptance flag is read a second time before the upgrade completes, and no session opens onto a pool that is shutting down. Controls the operator owns says what that identity is worth: it is attribution, and it scopes nothing.

A raw upgrade against a server whose hook rejects, and against a server that has stopped accepting, reads back:

rejected: HTTP/1.1 403 Forbidden
not accepting: HTTP/1.1 503 Service Unavailable

An accepted socket joins the live set, gets its ping bookkeeping and its optional age timer, and then carries a capnweb session over a fresh ConnectionApi. That RpcTarget implements ExecutorApi and holds the bridge settings, the instrumentation and that connection’s identity for as long as the socket lives.

Publishing the version is the server’s job, and enforcing it is the caller’s. protocolVersion() answers the PROTOCOL_VERSION constant of the protocol package. Session.open in packages/cloudflare-codemode/src/session.ts calls it before anything else crosses the socket and compares with exact equality through assertProtocolCompatible. Disagreement is a PROTOCOL_MISMATCH fault raised on the caller’s side at connect time, and a socket that never opened or an upgrade that was refused is a CONNECTION_REJECTED fault.

Bridge across two sessions

A caller’s dispatch function never leaves the caller’s process, and the guest that calls it runs two process boundaries away. Two independent capnweb sessions span that distance: the WebSocket session between the caller and the server, and the IPC session between the pool and one forked child. Their tables are separate, so no stub crosses from one to the other on its own. runBridgedExecution in packages/executor/src/server/bridge.ts copies each call from one session onto the other.

sequenceDiagram
    participant C as Caller
    participant S as Server
    participant P as Pool
    participant W as Child
    participant I as Isolate
    C->>S: execute with the dispatch and outbound stubs
    S->>S: validate the providers, duplicate the stubs, resolve the bounds
    S->>P: pool.execute with the resolved request
    P->>W: execute over the IPC session
    W->>I: fresh isolate, entry module
    I-->>W: __dispatch
    W-->>P: the same call over IPC
    P-->>S: the forwarding function
    S-->>C: the call over the WebSocket
    Note over S,C: a socket that died under this call answers the guest an error envelope rather than ending the run
    C->>S: envelope JSON
    S->>P: the same string back
    P->>W: over IPC
    W->>I: the proxy resolves
    I->>W: the run settles
    W->>P: result envelope
    P->>S: the same envelope
    S->>S: measure it against the response limit
    S->>C: result, or the RESPONSE_TOO_LARGE fault
    Note over S: the guard flips with the return, so a tool call arriving later answers the completed envelope

capnweb delivers a function passed as a call parameter as an RPC stub, and the session disposes that stub when the call it arrived on returns. The bridge asserts that what arrived carries dup, raising The dispatch function did not arrive as an RPC stub. when it does not. It then works through a duplicate of its own, so it disposes what it took and leaves the session’s copy alone. That duplicate rides a DisposableStack that releases on every exit path, the pool rejecting included, and the child entry keeps the same discipline for what reaches it over IPC.

Closed over that duplicate, a forwarding function reaches the pool, and it adds two things the caller’s own answer cannot carry. Calls that arrive after the run settled answer the pinned Execution has already completed. envelope and never reach a disposed stub. That guard flips before the duplicates die, because disposal runs in reverse order of registration. Transport failures under the call become an error envelope through describeFailure, so a dispatch whose connection died reaches the guest as a tool error instead of ending the run. But everything else passes through untouched: the arguments as one JSON string, the answer as the envelope the caller composed, and binary payloads still tagged. The protocol states that grammar, and the round trip of a tool call covers what the lower half of the path does with it.

Resolving the outbound function goes by the wire mode, not by its presence, because capnweb cannot tell an absent argument from an undefined one. stub duplicates the caller’s function and wraps it; disabled sends the pool null; native sends undefined, or null when the operator’s veto says so. Anything else is a PROTOCOL_MISMATCH fault naming the mode, so an unknown mode can never decay into native by accident.

Only a retained execution lets a duplicate outlive its call. When the result carries an executionId and the request arrived in stub mode, the bridge stops releasing that duplicate with the call and hands it to releaseWithHandles. That subscribes to the pool’s handles.close event and disposes the duplicate when that execution’s handles end. It checks hasHandles immediately after it is placed, because the close can land between the result and the subscription. In the child, a matching duplicate stays in its own map for the same span. So a handler served by invokeHandle keeps fetching through the caller that started the execution, under the terms of execution handles.

Bounds the server settles

Numbers reach the pool already settled. The bridge settles them, reading the caller’s request against the server’s configuration.

function resolveTimeout(requested: number | null | undefined, defaults: ExecutionDefaults): number {
  const bound = Math.min(defaults.maxTimeout ?? MAX_SUPPORTED_TIMEOUT, MAX_SUPPORTED_TIMEOUT)

  if (requested === null) {
    return bound
  }

  return Math.min(requested ?? defaults.timeout ?? DEFAULT_TIMEOUT, bound)
}

Requests that name a timeout get it capped at the ceiling; a request that names none falls to the server’s configured default and then to DEFAULT_TIMEOUT; a null timeout lifts the bound to the ceiling. That ceiling itself is capped at MAX_SUPPORTED_TIMEOUT whatever an operator configured. Past the widest delay a 32-bit timer accepts Node fires a timer almost at once, and a lifted bound would become no bound at all. Both constants are the protocol’s. But the grace and the isolate memory limit take no value from the request. They come from the server’s configured defaults, falling back to DEFAULT_GRACE and DEFAULT_MEMORY_LIMIT in packages/executor/src/executor/defaults.ts.

Provider validation happens here too. It runs ahead of the pool. Descriptor lists that cannot become the entry’s declarations come back as an error inside the envelope, and no worker is leased, no stub is duplicated and no child is touched.

The response limit is the server’s own bound on top of the payload bounds the protocol fixes. But it is absent by default. When it is set, the whole result envelope is serialized and measured in UTF-8 bytes before it reaches the session, logs included, because a run can outweigh its own result in console output. The run’s own value sits in that envelope as text, so its quotes and backslashes are escaped a second time and the count is larger than the value alone. A result above the bound is a RESPONSE_TOO_LARGE fault, not a truncated answer. Measuring precedes the send, because capnweb aborts the whole session when an incoming message passes the ceiling it puts on one message. That would cost every execution on that socket, and not just the one that overran. A grant that dies at the limit closes its own handles first: the caller never learns the executionId, so no one else could ever release the worker it holds. Handle invocations are measured with the same bound, their heads serialized as JSON and their bodies counted in bytes, since JSON.stringify cannot measure an ArrayBuffer.

Sockets carry bounds of their own, separate from the execution’s. maxConnectionAge, unset by default, schedules a timer that closes the connection with close code 1000 and the reason connection age limit. That timer is clamped to the same 32-bit bound and unreferenced, so it never holds the process open. Callers reconnect, and they redistribute over replicas after a scale-out that way. On pingInterval, the heartbeat sweeps the whole live set. A socket that has answered since the last round is marked unanswered and sent a ping, and a socket that has not answered is terminated. Turning the heartbeat off means a pingInterval of zero or less, which builds no interval at all. Anything above the 32-bit bound is clamped to it. The HTTP server holds a fixed 72 second keep-alive, with its headers timeout two seconds above it. Both are constants in listener.ts and neither is an option. That keep-alive clears the sixty seconds an Application Load Balancer holds an idle connection by default, so a balancer that reuses a connection this server is closing does not answer its own caller with a gateway error. But AWS takes that idle timeout as high as 4000 seconds, and a balancer set above 72 seconds needs the constant raised with it.

Health, readiness and the scrape

packages/executor/src/server/probes.ts serves the probe paths over the same HTTP server the upgrades arrive on. It claims a request only for GET and HEAD, because a scrape can cost the host real work. Every other verb falls through and the listener answers 404. Paths are read up to the query string, so a scraper that appends parameters is still served.

/healthz answers 200 with ok while the process is up. /readyz answers 200 with ready or 503 with not ready. One predicate decides it, supplied by the listener as accepting and not draining. It runs no execution and touches neither the workers nor the queue, and each of those is a decision, not an omission. Readiness gated on a live worker would never recover on a fresh replica. Spawning happens on demand, and the traffic that creates the first worker is exactly what readiness withholds. Readiness gated on the queue standing at its bound takes every replica out of the Service at once under an even load. That turns a typed, retryable fault into no address at all. But the queue reaches a caller as the QUEUE_OVERFLOW fault, and an operator as the queue-depth gauge an autoscaler reads.

/metrics is served only while a metrics source is configured and stays unclaimed otherwise, so an unconfigured server answers 404 and not an empty scrape. But a render that fails is logged and answered 500 with metrics unavailable, so a collector that is down cannot leave the request hanging.

Metrics seam

Executor packages measure themselves through @opentelemetry/api and register nothing. Without a registered SDK the API hands out no-op instruments, so an embedding host that wants no measurement pays almost no cost, and a host that wants it decides where the numbers go. That division is OpenTelemetry’s own guidance to library authors: a library depends on the API, and only the final application installs an SDK. Scrapes follow the same rule. ServerOptions.metrics takes anything that renders text:

export type MetricsSource = {
  readonly contentType: string
  render(): Promise<string>
}

The server holds no exporter behind that seam, and the specs prove it against the real @opentelemetry/exporter-prometheus. That package is a development dependency here, not a shipped one. instrumented: false builds no instruments at all. silentInstrumentation replaces the counters and the span with functions that do no work, and the pool carries the same switch.

Server measurements cover the part of a request the pool never sees. executor.connection.active rises and falls with the live set, and it falls only when the set actually held the socket, so a second close event cannot drive it below the truth. executor.connection.rejections counts what the hook turned away. Every execute runs inside an active executor.request span carrying executor.identity, so the pool’s own spans nest under it wherever the host registered a context manager. Telemetry names every instrument and what a Prometheus scrape renames it to.

Embedding the server

createExecutorServer is the whole surface: one options object in, one handle out.

import { createExecutorServer } from '@supolka/codemode-executor/server'

const server = createExecutorServer({
  host: '127.0.0.1',
  port: 8080,
  onConnection: ({ headers, remoteAddress }) =>
    headers.authorization === secret
      ? { identity: remoteAddress }
      : { reject: true, reason: 'the authorization header did not match' },
  executor: {
    pool: { maxWorkers: 4, maxQueue: 200 },
    outboundMode: 'null',
    defaults: { timeout: 30_000, maxTimeout: 120_000, memoryLimit: 256 },
  },
  responseLimit: 4 * 1024 * 1024,
  shutdownDelay: 5000,
  logger,
})

await server.listen()

listen() resolves when the port is open and rejects with whatever the HTTP server emitted, so an address already taken reaches the host as its own EADDRINUSE and not as a wrapped fault. socketPath replaces the host and port with a unix socket, and address() is the HTTP server’s address. A host that asked for port 0 learns the port it got from there. That handle also exposes the WorkerPool itself, so the events the pool publishes are available without a second subscription seam.

The server leaves the rest of a process to the host. It installs no signal handler, registers no OpenTelemetry provider, opens no second port and takes no setting from the environment. The environment reaches the pool in the default it hands a forked child and in the CODEMODE_TESTING switch the threat model accounts for, and nowhere else. A metrics endpoint exists only while metrics names something that renders text, and every other decision a deployment makes sits in the options object above. startDaemon is one set of answers to those questions, not the only one.

Where a setting comes from

commander owns the command line and zod owns the shape. Each setting is declared once in apps/daemon/src/cli/options.ts with its flags, its environment variable, its description and its default. Commander binds the variable to the flag, so a flag beats a variable and a variable beats the default. The daemon reads that provenance back out of commander, and the settings block of check prints it:

--max-workers is 1, from the flag.
--max-queue is 42, from the environment.

zod then parses the strings a command line and an environment can only carry as text, and refuses the values that would leave the daemon unable to serve. A zero for the workers, the executions per worker, the queue, the retirement count, the memory limit or the timeout parses as a whole number. It would leave a daemon with no worker, no execution slot, a queue that is full from the start, a child retired before its first execution, no heap or a run that expires on arrival. So the schema names 1 as the floor for each. Durations past the widest a timer accepts are refused for the reason the clamp above exists. So is a shutdown delay that is not smaller than the shutdown grace, because the delay is spent inside the grace, not added to it. Every problem is reported at once, each line naming both spellings of the setting it is about, in the shape --max-queue or MAX_QUEUE: Expected at least 1.

buildServerOptions in apps/daemon/src/config.ts is the only translation from that configuration to ServerOptions, and an absent metrics source turns instrumentation off in the pool and the server together. It reaches maxWorkers, executionsPerWorker, maxQueue, retireAfterExecutions, the logger and whether the pool measures anything. But the rest keep the pool’s own defaults, so the idle bound on retained handles, the worker memory bound behind retirement and the termination delay behind a hung child are not an operator’s to set.

Composing the daemon

startDaemon in apps/daemon/src/daemon.ts is the composition root. Telemetry is registered first, since the providers must exist before anything measures. Then the executor server listens, the pool counters subscribe to its events, and the admin app listens if it is turned on:

flowchart LR
    subgraph starting["taken in this order"]
        direction TB
        t1["telemetry providers"] --> s1["executor server listens"] --> c1["pool counters subscribe"] --> a1["admin app listens"]
    end
    subgraph stopping["released in this order"]
        direction TB
        s2["server drains, then closes"] --> a2["admin app closes"] --> t2["telemetry flushes"]
    end
    starting -. stop .-> stopping

Whatever the start took is held on one AsyncDisposableStack, and the daemon takes ownership of it through move() once the start succeeded. A step that fails disposes what came before it, so a failed start leaves nothing listening. But the protocol server is deferred last, both because it must be released first and because the window between its listen and its registration is covered explicitly in the failure path. Releasing runs in the inverse of the registration order, the same order a drain wants: the protocol port spends its delay draining while the admin app still answers, and the operator keeps status and maintenance for the whole shutdown.

Pool counters are a subscriber like any other. countPool derives the worker, execution, queue and rejection counts from the pool’s own event emitter, so the status an operator reads is counted even in a process that registers no SDK and measures nothing.

The admin app is a hono app on a second listener, bound to the loopback address by default. It serves the process status at /__status__, the settings in force with the source of each at /__config__, and the maintenance switch as GET, PUT and DELETE on /__maintenance__. An unknown route answers a JSON 404 reading {"error":"No such route."}, and a failure inside the daemon answers a JSON 500 and not a stack. status and maintenance on the command line are a typed hono/client built from typeof app, so the two halves cannot drift. Every call that client makes carries a five second deadline, so a wedged daemon cannot hang the command. Maintenance drives the server’s acceptance switch, so readiness answers 503 while the executions already accepted run to their end.

startTelemetry in apps/daemon/src/telemetry.ts composes the pipeline explicitly, and not through the SDK’s own bootstrap. A Prometheus reader is registered whatever else is configured, so a scrape answers without an operator setting anything up. It reaches the listener through the MetricsSource seam, not through a second HTTP server. OTLP is added per signal when an endpoint variable names a collector, and registering the tracer provider also installs the context manager the executor’s spans nest under. stop shuts both providers down together and settles them even when one fails, so a collector that is down cannot restate a clean drain as a failed one. It disables the API globals afterwards, so the same process can start another pipeline. OTEL_SDK_DISABLED short-circuits all of it, and a daemon started that way answers /metrics with 404 while readiness still answers 200.

Stopping

Two durations bound a stop. One nests inside the other. The shutdown delay lives inside the server’s close(), and the shutdown grace bounds the daemon’s whole disposal.

sequenceDiagram
    participant K as Platform
    participant D as Daemon
    participant S as Server
    participant P as Pool
    K->>D: SIGTERM
    D->>S: close
    S->>S: draining, readyz answers 503
    K->>S: readyz, healthz, a new execution
    S->>K: 503, 200, and the execution runs
    Note over S: the shutdown delay ends here
    S->>P: shutdown
    S->>K: close 1001 on every socket
    D->>D: admin app closes, telemetry flushes
    Note over D: the shutdown grace bounds all of the above
    D->>K: exit

close() sets the draining flag first. Readiness answers 503 from the first moment, while the listener keeps accepting for the whole delay. An endpoint controller needs that window, because it stops routing asynchronously and a caller it routed just before the flip would otherwise meet a closed port. When the delay ends, acceptance stops, the heartbeat interval is cleared, the pool shuts down, every open socket is closed with close code 1001 and the reason server shutting down, and the HTTP server closes. That promise is memoized, so a second close() cannot start a second drain.

Daemon.stop races the disposal of the whole stack against the shutdown grace and answers whether it finished. That answer travels on the stopped event. The command line installs close-with-grace with the same grace and calls stop from its handler, so a stop that finishes exits the process cleanly. But a stop that outlives its grace is ended by close-with-grace instead of hanging. A daemon polled across a SIGTERM shows the shape: readiness turns 503 within milliseconds while liveness and the admin app keep answering, the listener stays up for the configured delay, and the process exits 0.

Container image and the manifests

apps/daemon/Dockerfile has a builder and a runtime stage, and both take the same ARG NODE_IMAGE. isolated-vm is compiled from source in the builder, and its binary has to meet at runtime the glibc it was compiled against. A stage naming its own base can produce a container that fails to load the engine, with only a linker error to read. Debian is the base this repository builds and supports, and it makes no claim about musl.

The builder installs corepack globally, since Node stopped shipping it in version 25. Corepack then reads the pnpm version out of the repository’s own packageManager field, not a second copy pinned in the Dockerfile. The builder also installs the toolchain that compilation needs. The patch this repository carries changes isolated-vm’s C++, and the binaries the package ships were built without it. Copying the lockfile, the manifests and the patches before the sources lets the install layer survive every change to the code. tsc --build runs against the daemon’s own project references and not nx, because the build context carries only the projects the daemon needs while nx checks the whole workspace. pnpm deploy --prod then writes the daemon, its production dependencies and the workspace packages it uses into one self-contained directory. The compiled engine is stripped of its debug sections, and its intermediate objects and foreign prebuilds are deleted, since node-gyp-build looks in build/Release first and would never load them. Its last step runs the daemon’s own check inside that directory, so a binary the stripping damaged fails the image build, not every container started from it.

The runtime stage copies that directory owned by node and runs as the unprivileged node user. A symlink puts the entry point at /usr/local/bin/codemode-executor, so an operator can ask a running container for its status by name. Its HEALTHCHECK polls /healthz on the port the environment names, and docker ps and compose read that. Kubernetes polls the same path through its own probe.

Manifests under deploy/ are a deployment that has already made the operator’s decisions, and apps/daemon/src/deploy.spec.ts holds them to the daemon they run. Every variable they name is either a declared setting or an OTEL_ one, since a variable nobody reads is a setting an operator believes they have made. Both probes point at the port the container actually listens on, resolved through the named port and not a repeated number. Their termination grace period exceeds the shutdown grace, because SIGKILL arrives when it runs out. The memory limit covers three times the product of the workers, the executions per worker and the isolate limit: the upper end of the two to three times overshoot isolated-vm documents for a bound it calls a guideline. On the compose side the published port matches PORT, the stop grace period exceeds the shutdown grace, and the Dockerfile the compose file builds is the daemon’s own file byte for byte. A separate case starts a real daemon and asks it for every path the manifests and the image poll.

Check command and the exit codes

codemode-executor check is the daemon’s answer to whether a machine can serve at all. runCheck in apps/daemon/src/check.ts builds an IsolateExecutor with the configured timeout, memory limit, worker count and queue bound. One job then crosses the whole path an execution needs: a forked child, a fresh isolate, a timer on the host and an answer that arrives only after the isolate yields. It demands the answer 42. A sandbox that answers something else fails the check like one that never started. But nothing escapes it as a throw. A machine that cannot serve comes back as a result carrying a failure string. That lets --json answer a machine, and the human form print the reason with the settings that are not at their default.

Exit codes are declared in apps/daemon/src/cli/index.ts and printed under Exit codes in the help. A clean stop is 0. An address another process already holds is 2. exitCodeFor recognizes it by the EADDRINUSE code Node sets on the error, not by its text, because that is the one start failure an operator fixes somewhere other than the settings. Every other failure runCli catches is 1, a setting the schema refused among the rest. A check that ran and reported that this machine cannot run sandboxed code is 5, and a container crash-looping on such a machine reports that code while its command is check.

This socket has a client at its other end. The adapter is the client this repository ships, and the dialect it serves on top.