Worker pool

The sandbox runs one request and disposes the isolate it ran in. Nothing inside it knows that a second request exists, which process it is running in, or which of two arrivals goes first. WorkerPool keeps a bounded set of child processes, gives each request the least loaded one with room, and replaces the children that die or wear out. Requests past the last worker wait in a bounded queue, and execute rejects at once whatever arrives at a full one.

All of it lives in packages/executor/src/pool. Two thin layers sit on top and neither adds machinery of its own. IsolateExecutor in packages/executor/src/executor/isolate-executor.ts implements the native contract inside the host’s own process. The per-execution bridge in packages/executor/src/server/bridge.ts implements the same contract for a caller over a WebSocket, and that bridge is the next chapter. The executor protocol states what a caller may ask for, the operator guide what a deployment decides, and the threat model what each bound is claimed to hold.

Each worker is a forked process and its watcher

Each worker is one forked Node process and the object that watches it. Worker in supervisor.ts holds the child, the capnweb session over its IPC channel, the health watch and the last memory reading the child sent. It lives from the first answered heartbeat until the child exits, stops answering or is shut down. Every death outside a shutdown is published as a crash.

Worker.spawn() forks childEntryPath with the execArgv, env and cwd it was given, through child_process.fork. That same call opens the single IPC channel every exchange below travels over. isolated-vm cannot create an isolate in a process started from the Node startup snapshot, so the pool passes --no-node-snapshot to every child by default. assertNoNodeSnapshot in sandbox/isolate.ts tests that by building a trivial isolate, not by reading execArgv back. Before it awaits anything, spawn opens an RpcSession over IpcTransport on the fresh channel and takes the child’s remote main, typed as ChildApiShape.

Then spawn requests one heartbeat under spawnTimeout instead of the heartbeat deadline, because booting a process takes a different scale of time than probing one that is already running. A heartbeat that never arrives rejects the spawn with WORKER_CRASHED, retryable, reading Worker <id> failed to start. with the underlying failure as its cause. An answered one marks the worker spawned, starts the health watch, and only then emits spawn with the child’s pid, so a listener that throws cannot leave a live child running unwatched. Liveness is one field: alive reads true while the worker still holds the remote stub, and a shutdown and both routes to death clear it.

Startup order in the child

child-entry.ts acts only when it is the fork’s own entry script. runsAsEntryScript compares import.meta.url against the realpathed process.argv[1], because the ESM loader realpaths the main module and a symlinked entry would otherwise never match. Imported inside a test runner the module stays inert, so no stray session opens in a process that merely happens to load it.

new IpcTransport(process) attaches the message listener before anything is awaited, so a frame the parent sends while the module is still evaluating reaches a channel somebody listens on. That transport asserts an open IPC channel at construction and throws INTERNAL without one: a transport over anything else is a wiring bug, not a runtime condition. new ChildApi() then composes one Sandbox with a cachedDataStore map. One sandbox per child lets the compiled-module cache and the startup snapshot outlive an execution, while each execution still gets an isolate of its own.

assertEngineServesSandbox is another field of ChildApi, so constructing the class starts the engine check and stores its promise without awaiting it. heartbeat awaits that promise before it returns anything. That puts the verdict exactly where the supervisor decides a worker is live. A child whose isolated-vm build cannot serve this sandbox never answers a first heartbeat, its spawn rejects with WORKER_CRASHED, and the text naming the missing behaviours travels on the chain of causes behind that rejection.

The engine check runs inside spawnTimeout and not beside it. assertEngineServesSandbox runs its three probes together under Promise.all, and answerWithin bounds a whole probe instead of the calls inside it, so the check spends one ENGINE_PROBE_TIMEOUT of 5 s and not their sum, inside the 10 s spawnTimeout defaults to. Neither number is an option the pool exposes, so moving either one is an edit to sandbox/isolate.ts or supervisor.ts. A boot that was merely slow on a loaded machine reaches the same rejection by the other route: its first answer arrives past spawnTimeout, and the supervisor reports it as a worker that failed to start.

How a request finds a worker

WorkerPool.execute stamps the arrival time, rejects with INTERNAL and the text The pool is shutting down. when the pool is going down, mints an execution id from a counter, and leases a worker. The wait it measures is the gap between arrival and lease, and that number travels on the execution.start event.

leaseWorker publishes the queue depth before anything else. Every execution announces the depth it found, so a pool that has never queued still exports the series and a fresh replica can be scaled on from its first execution. Then it looks for room:

flowchart TD
  request["execute(request)"] --> depth["publish queue.depth"]
  depth --> pick{"an alive, undrained worker with capacity"}
  pick -->|"the least loaded, earliest on a tie"| lease["count the load, run on it"]
  pick -->|"none"| room{"workers + spawning below maxWorkers"}
  room -->|"yes"| spawn["spawn, then lease the new worker"]
  room -->|"no"| queue{"the queue is below maxQueue"}
  queue -->|"yes"| wait["enqueue the lease and wait"]
  queue -->|"no"| overflow["reject with QUEUE_OVERFLOW"]
  spawn -->|"the first heartbeat is answered"| lease
  spawn -->|"no answer within spawnTimeout"| failed["reject with WORKER_CRASHED"]
  wait -->|"serveQueued, after a settle, a retirement or a respawn"| lease

Every refusal leaves execute before runOn, so it publishes execution.reject and no execution.start. That keeps work turned away apart from work that ran and failed. Four refusals reach it: the overflow, a spawn that never answered, and the two that can reach a lease already waiting, a shutdown draining the queue and a replacement spawn that keeps failing.

WorkerLoad in queue.ts counts running executions per worker against executionsPerWorker. pickLeastLoaded walks the candidates, skips every worker already at its capacity, and keeps the earliest of equal counts, because the comparison is strict and the caller’s order is therefore the tiebreaker. Candidates are the workers that are alive and not marked for a drain. start raises the count and throws INTERNAL when the worker is already full, so a broken dispatch loop fails at once instead of overloading a child unnoticed.

With no candidate that has room, and workers.size + spawning.size under maxWorkers, the pool spawns a new child. Counting the spawns in flight keeps two concurrent leases from spawning past the bound.

Past both, the caller waits. Each lease is enqueued as its promise’s resolve and reject, and that promise stays pending until serveQueued hands it a worker. serveQueued runs after every settle, after a retained execution frees a slot, after a retirement and after a crash respawn, and each time it resolves as many waiting leases as it finds candidates for. With no candidate it spawns one where the bound allows and the pool is not shutting down. A successful spawn serves the queue again. But a failed one goes to failOneLease, which rejects a single waiting lease and serves again, so a spawn failure that persists drains the queue instead of parking the rest of it forever. failOneLease passes a typed fault through untouched and wraps anything else as WORKER_CRASHED, retryable, reading A worker failed to spawn.

Queue and its bound

BoundedQueue in queue.ts is a first-in-first-out list with a hard bound, maxQueue. enqueue past the bound throws QueueOverflowError from the protocol package, and the throw happens inside the promise executor of leaseWorker, so the caller’s own execute rejects with it. Three concurrent runs against one worker and a queue of one leave the third caller with no room:

import { ExecutorError, IsolateExecutor } from '@supolka/codemode-executor'

const executor = new IsolateExecutor({ pool: { maxWorkers: 1, maxQueue: 1 } })

const busy = `async () => {
  await new Promise((resolve) => setTimeout(resolve, 300))

  return 'done'
}`

const settled = await Promise.allSettled([
  executor.execute({ code: busy, outbound: null }),
  executor.execute({ code: busy, outbound: null }),
  executor.execute({ code: busy, outbound: null }),
])

console.log(
  settled.map((outcome) => {
    if (outcome.status === 'fulfilled') {
      return outcome.value.result
    }

    const refused = outcome.reason as ExecutorError

    return { code: refused.code, retryable: refused.retryable, message: refused.message }
  }),
)

await executor.dispose()

[
  'done',
  'done',
  {
    code: 'QUEUE_OVERFLOW',
    retryable: true,
    message: 'The queue is at its bound of 1. Retry when a running execution settles.'
  }
]

retryable is true and the message names the bound the caller hit, so the caller has something to retry against and an operator a number to scale on. An unbounded queue would take the same third request, report nothing, and hold it in the parent’s memory for as long as the load lasted.

queueFull reads true while the queue is at its bound and false again as soon as a dequeue frees room. Shutdown empties the queue from the other end. Every waiting lease is rejected with INTERNAL and the same The pool is shutting down. a fresh execute meets, retryable because a queued execution never started, and the new depth is published once for all of them.

Lifecycle of a worker

Fresh isolates cover what leaks inside one execution. What outlives an execution accumulates in the child process instead, and shouldRetire in lifecycle.ts measures that growth against two bounds.

stateDiagram-v2
  [*] --> Spawning: fork the child entry
  Spawning --> Serving: the first heartbeat is answered
  Spawning --> [*]: no answer within spawnTimeout, the spawn rejects
  Serving --> Draining: shouldRetire returns the reason executions or memory
  Draining --> Retired: the running count reaches zero
  Retired --> [*]: SIGTERM, then SIGKILL after the grace
  Serving --> Dead: exit, an unanswered heartbeat or a broken session
  Draining --> Dead: exit, an unanswered heartbeat or a broken session
  Dead --> [*]: SIGKILL, a replacement is spawned

shouldRetire compares the worker’s execution count against retireAfterExecutions and its memory reading against workerMemoryLimit. It tests the execution count first, so a worker over both bounds retires with the reason executions. Both comparisons include the bound itself. That memory reading is the child’s own process.memoryUsage.rss() from the last heartbeat, compared as it arrives, with one unit on both sides and no scaling anywhere in the module.

retireWhenDue runs on a live worker after every settle, and after a retained execution frees its slot. It raises the pool’s own count for that worker, calls shouldRetire, and marks the drain when the decision is to retire. Load settles first, so the verdict already counts the execution that just finished.

WorkerDrain records the marks and nothing else. mark stores the reason and the first mark wins, so a second cause does not change it. isMarked takes the worker out of the lease candidates, so a draining worker gets no new work, and drained returns true once a marked worker’s running count reaches zero. Nothing in flight is cut short for a retirement.

retireWorker drops the worker from the pool’s maps first, publishes worker.retire with the reason, the execution count and the memory reading, then shuts the child down with the ordinary grace. Where the pool is not shutting down and the bound allows, it spawns a replacement, and it serves the queue either way. Replacements that fail to spawn are logged and routed into failOneLease. A pool of one worker retiring after two executions publishes the whole sequence:

import { WorkerPool } from '@supolka/codemode-executor/pool'

const pool = new WorkerPool({ maxWorkers: 1, retireAfterExecutions: 2 })

pool.events.on('worker.spawn', (event) => console.log('worker.spawn', event.workerId))
pool.events.on('execution.settle', (event) =>
  console.log('execution.settle', event.executionId, 'on worker', event.workerId, event.outcome),
)
pool.events.on('worker.retire', (event) =>
  console.log('worker.retire', event.workerId, 'after', event.executions, event.reason),
)

for (const run of [1, 2, 3]) {
  await pool.execute({
    code: `async () => ${run}`,
    providers: [],
    timeout: 60_000,
    grace: 1000,
    memoryLimit: 128,
    outbound: null,
  })
}

await pool.shutdown()

worker.spawn 1
execution.settle execution-1 on worker 1 success
execution.settle execution-2 on worker 1 success
worker.retire 1 after 2 executions
worker.spawn 2
execution.settle execution-3 on worker 2 success

execution-3 runs on worker 2, because the retirement dropped worker 1 from the maps before the third execute looked for a candidate. The worker.retire line names executions as its reason, so the count bound fired and not the memory bound, and the execution ids come from the pool’s own counter.

Termination ends a child no bound inside the isolate can reach: one wedged in native code, or anywhere else the engine’s own timers do not run. runOn schedules one termination per execution, at the run’s timeout plus its grace plus terminationDelay, clamped to 2 ** 31 - 1. When it expires it logs terminating a hung worker and sends the child SIGKILL. The clamp is there because a larger sum reaches setTimeout past the widest delay a 32-bit timer accepts, where Node sets the delay to one millisecond and fires almost at once. Settling cancels the schedule in the finally of runOn, and shutdown cancels every pending one.

Host waits pause a termination. The child announces the run clock’s own edges over the session through reportHostWait, pause freezes the remaining delay, and resume schedules what is left. runOn gives a request that counts host waits a null reporter instead, so that run keeps the plain wall-clock bound. Without the pause, a termination at the timeout plus the grace plus the delay would kill exactly the runs a stopped clock is meant to keep alive. The sandbox chapter says where those edges open, and Timeouts and the run clock states the semantics they follow.

When a child dies

Every way a child can die arrives at Worker.fail. The child’s exit event reports child exited with code <code> and signal <signal>. An unanswered heartbeat reports its own timeout text. That heartbeat is the signal that catches a hung child, because a busy loop never exits by itself. A broken session under an in-flight call reports what the transport said, and it arrives first, because the session breaks before the exit event does.

fail is idempotent and the first reason wins. It clears the remote, stops the health watch, sends SIGKILL so a hung child cannot outlive its worker, and emits crash for a worker that was announced. A child that never came up publishes no crash, because the spawn rejection already reached its caller.

Executions under the dying child reject with WORKER_CRASHED, retryable, reading Worker <id> crashed during an execution: <reason>. But two failures take other routes through wrapExecuteFailure. A rejection with a typed code other than CONNECTION_BROKEN passes through untouched, because the child answered and lives on. Stub mode with no outbound function is the case that produces one: the child raises PROTOCOL_MISMATCH, and the worker keeps serving after it. An execution cut by a shutdown reports CONNECTION_BROKEN and the text Worker <id> was shut down during the execution., so the pool does not retry against a worker whose shutdown it started.

handleWorkerCrash does the pool’s half. It builds the crash event with the ids of the executions that were running on that worker, and announces the close of every retained execution the child held with the reason crash. Dropping the worker from the maps resets its load and clears its drain mark. Then it shuts the child down with no grace and spawns a replacement, unless the pool is shutting down. Publishing comes last, after the bookkeeping. Other executions sharing that worker meet the same WORKER_CRASHED through their own broken calls, since the session rejects each of them. Queued leases are untouched and wait for the replacement.

Heartbeat

startHealthWatch sets a heartbeatInterval timer, and each tick requests a heartbeat. requestHeartbeat races that call against a heartbeatTimeout deadline, and both timers are unreferenced so neither holds the parent process open on its own. A heartbeat that loses the race declares the death through fail and rejects with WORKER_CRASHED. startHealthWatch swallows that rejection, because the declaration has already happened and the rejection adds no new fact.

Only the child can measure what comes back. ChildApi.heartbeat in child-entry.ts awaits the engine check, then returns the three numbers of ChildHeartbeat: memory, the child’s resident set size from process.memoryUsage.rss(), running, the executions and handle invocations inside it right now, and executions, the count of runs that reached the sandbox. Calls refused before any accounting never reach that last count, stub mode without a stub among them.

Each worker stores the memory reading in lastMemory, and worker.memory returns it to the retirement check. But the pool keeps a per-worker execution count of its own in executionCounts and retires on that one, not on the child’s.

IPC transport and the message ceiling

Pool and children speak capnweb over the fork’s IPC channel. IpcTransport implements capnweb’s RpcTransport over an endpoint that is process in the child and the ChildProcess in the parent. A forked pair always shares one channel, so an endpoint without one fails at construction, not at the first send.

A frame is one string. send hands it to endpoint.send with a callback. A channel that closes mid-send raises an error event on the endpoint when nothing takes the callback, and an error event nobody listens for is fatal, so the callback routes that race into the ordinary disconnect path. receive resolves with the next string, and only strings belong to the session, so anything else on the channel is someone else’s traffic and is ignored. Frames arriving with no receiver waiting are queued, and a disconnect drains the queue before it starts rejecting, so the session reads everything the peer sent before it died. Node’s IPC serialization carries whole strings unchanged, so capnweb frames need no framing of their own on top of it.

The ceiling on one message belongs to capnweb, not to the channel underneath it. DEFAULT_LIMITS.maxMessageSize sets it at 33 554 432 UTF-16 code units. Each session measures an incoming frame against it in its read loop, before the JSON is parsed. A frame over the bound throws out of that loop and takes the whole session down with it. listener.ts opens the caller’s own session with newWebSocketRpcSession and passes no limits, so the same ceiling governs the WebSocket in front of the server. Underneath it, the IPC channel adds no bound at that scale.

ipc-transport.ts repeats that bound on the sending side, because the session that would refuse an oversized frame is the peer’s. Left alone, the parent would watch its child’s session die on a TypeError about an incoming message, and that reads as a worker dying for no stated reason. So send fails the transport first with a CONNECTION_BROKEN naming the size of the frame it refused. But every payload a guest builds is bounded far under that where it is built, at the PAYLOAD_LIMIT and CONSOLE_CAPTURE_LIMIT of the bound on everything a run sends. A frame that reaches this check is a defect in this repository, and the check exists so the defect names itself. MESSAGE_LIMIT in ipc-transport.ts is a copy of capnweb’s ceiling, and no lane compares the copy against DEFAULT_LIMITS, so a change upstream leaves it stale in silence.

One session runs in both directions. The parent holds the child’s remote main, and a function passed as a call argument crosses as a live stub the child calls back through. Tool dispatch, the outbound function and the host-wait reporter reach the parent that way. protocol.ts states the rest: the request is data alone, and the tri-state outbound rides beside it as an explicit mode, so no meaning hangs on how the wire treats an empty slot. runOn builds that mode from the request’s outbound field, mapping an absent one to native, null to disabled and a function to stub, and it passes the function itself beside the request.

Live objects cross that session, and capnweb’s reader changes them on the way in. evaluateImpl walks an incoming object and deletes every own key that is in Object.prototype, along with toJSON. Upstream states the reason beside the loop: stopping a peer from overriding an Object.prototype member, __proto__ above all, and stopping a toJSON from snooping on the session’s own JSON.stringify. But neither a raised error nor a log line marks it. { ordinary: 1, constructor: 2, toString: 3, valueOf: 4 } serializes to exactly that JSON and arrives as { ordinary: 1 }.

So this protocol sends no plain object whose keys a caller chose. A run’s result crosses as resultJson, the JSON text the isolate produced, and IsolateExecutor parses it with parseForDispatch. A request’s module map crosses as [name, source] pairs built by listModuleEntries, which also turns a json source into the module source it stands for, and the child rebuilds the map with Object.fromEntries. Tool calls were never exposed, because the dispatch grammar already sends the arguments and the answer as one JSON string each. Both directions survive the round trip a deleted key would spoil:

import { IsolateExecutor } from '@supolka/codemode-executor'

const executor = new IsolateExecutor({
  modules: { toString: `export const from = 'a module named toString'` },
})

const { result } = await executor.execute({
  code: `import { from } from 'toString'
async () => ({ ordinary: 1, constructor: 2, toString: 3, valueOf: 4, from })`,
  outbound: null,
})

console.log(result)

await executor.dispose()

{
  ordinary: 1,
  constructor: 2,
  toString: 3,
  valueOf: 4,
  from: 'a module named toString'
}

constructor comes back as 2, so the run’s own object crossed as text and not as a live object, and from resolving proves the same of a module map keyed on toString. The daemon hop carries both the same way, because resultJson and the module entries are text on that session too and ExecuteResultMessage names no Object.prototype member of its own. That deletion is a defect in capnweb, and Dependencies says the fix goes there.

Round trip of a tool call

Tool calls leave guest code and reach the caller’s own function across one isolate boundary and one or two process boundaries, and the answer comes back the same way:

sequenceDiagram
  participant G as guest code
  participant C as child process
  participant P as pool parent
  participant H as the caller
  G->>C: __dispatch(provider, tool, argsJson)
  C->>C: measure the three strings against PAYLOAD_LIMIT
  C->>P: capnweb frame over the IPC channel
  Note over C,P: the run clock and the termination pause on this edge
  P->>H: the caller's dispatch, or the bridge's stub over the WebSocket
  H-->>P: one JSON string carrying the envelope
  P-->>C: capnweb frame
  C-->>G: ['returned', json] copied into the isolate

The call starts at the generated proxy. proxy-source.ts builds one Proxy per provider whose get trap turns a tool name into an async function, so the call becomes __dispatch(provider, tool, argsJson) with the arguments serialized by the dispatch codec. In the child, installDispatch in sandbox/dispatch.ts puts a host function behind __dispatch over an ivm.Reference, and that function applies the run’s own rules: the completion guard, the PAYLOAD_LIMIT measurement and the not-found envelope. It returns a settled two-element frame in every case. The sandbox chapter states each rule and the reason a rejection would be wrong there. This chapter picks the call up where it leaves the child process.

The host function calls the stub the parent passed as an argument of execute, wrapped by wrapHostWait, so entering it opens the run’s host wait and the parent’s termination pauses. That stub call crosses the IPC channel as a frame. In the parent, the function behind that stub is the caller’s dispatch wrapped by measureDispatch, which records executor.tool.duration around the round trip.

For IsolateExecutor the journey ends in the host’s own process, one process boundary away from the isolate. For a daemon it goes one further: the parent’s function is forwardDispatch from server/bridge.ts, which calls the caller’s dispatch stub over the WebSocket session. That session and the pool’s IPC session are separate capnweb sessions with independent tables. So a caller’s stub never crosses between them on its own, and the bridge owns the lifetime of the duplicate it holds.

An outbound request follows the same shape through a different host function. Sandbox fetch reaches __sendRequest, installed by sandbox/runtime.ts over a reference, and it calls whatever chooseOutbound picked for the mode. In stub mode it picks the parent’s outbound stub, wrapped by wrapHostWait as well, and the request travels as plain data the caller’s own policy sees. In native mode the child performs the request itself through the process’s own fetch, so nothing crosses the pool boundary, no host wait opens and the run’s clock keeps counting. In disabled mode the rejection is composed inside the child and never leaves it.

Retained executions and their slots

A run that granted handles comes back with an executionId in its result. runOn reads it, records the execution in retained against the worker that served it, and then skips two things it does for every other run. It does not settle the load, and it does not run the retirement check. That worker keeps counting the execution, because the run has settled while the child still holds a live isolate, and only the close puts the slot back into rotation. With one execution per worker, a retained execution occupies a whole worker. So the operator guide tells a deployment that grants handles to count those executions in its worker budget.

invokeHandle finds the entry, raises its running count, clears the idle timer and forwards to the child. But an id the pool does not know throws HandlesClosedError before any of that, and so does closeHandles. That idle bound watches the gaps between invocations, not a running one, because an invocation whose clock is paused on a caller-served wait can legitimately outlast it. When the count returns to zero, the timer is set again on handlesIdleTimeout. It is unreferenced, so an expiry cannot keep an otherwise finished host process alive.

events.ts names a close reason for each way a retained execution can end. closed is an explicit closeHandles. expired is the idle bound. terminated is an invocation that died at a run bound: the isolate is disposed, because disposal is the only way to stop it. crash is the child dying under it. shutdown is the pool going down, and there the closes are announced before the children die, so whoever holds a resource for a retained execution can release it. The bridge releases the caller’s duplicated outbound stub on exactly that event.

All of them go through releaseRetained, which deletes the entry, clears the idle timer and publishes handles.close. It is idempotent, so racing closers publish one event. A close, an expiry and a termination then call freeRetainedSlot, which settles the load, runs the retirement check and serves the queue. But a crash and a shutdown call none of it, because the worker itself is going and dropWorker resets its load with the rest of its bookkeeping.

terminated arrives differently from the rest. The child returns a settled frame for an invocation instead of rejecting it. Only the child can tell a handler’s own throw from a bound that disposed the isolate, and the pool has to learn which of the two ended the handles. A typed fault crosses as its code beside the text, set from an instance the child itself constructed, so a handler throwing the composed closed-handles words does not pass for the fault. Execution handles states the lifecycle a caller sees.

Shutting the pool down

pool.shutdown(options) works in a fixed order. It raises the shutting-down flag, so a fresh execute rejects before it can lease anything. It drains the queue, rejecting every waiting lease. Then it waits up to drainTimeout for every execution and every handle invocation still in flight, reading the counters every 25 ms because a settled invocation publishes no event of its own. An idle retained isolate is not waited for, and the per-execution termination schedules stay live through the wait, so a hung run is ended at its own wall instead of holding the drain to its bound. An invocation arriving after shutdown began throws HandlesClosedError before the lookup, whatever the drain is still settling, so only the invocations already in flight settle inside it. Past that wait it cancels every pending termination. Every retained execution is then released with the reason shutdown, while the pool is still whole enough for a subscriber to act on the announcement. It waits on the spawns in flight, so a fork that lands mid-shutdown cannot orphan its child. Each worker is shut down with the grace: the remote goes first, then SIGTERM, then SIGKILL when the grace runs out. Then it drops the workers, detaches the instrumentation and removes every listener.

ShutdownOptions carries both numbers. drainTimeout defaults to zero, so a bare shutdown() cuts every running execution at once. grace defaults to DEFAULT_SHUTDOWN_GRACE, one second. IsolateExecutor.dispose() calls shutdown() bare, so a run still working when dispose() reaches that call is cut where it stands:

import { ExecutorError, IsolateExecutor } from '@supolka/codemode-executor'

const executor = new IsolateExecutor({ pool: { maxWorkers: 1 } })

const running = executor.execute({
  code: `async () => {
    await new Promise((resolve) => setTimeout(resolve, 30_000))

    return 'finished'
  }`,
  outbound: null,
})

const outcome = running.catch((failure: unknown) => {
  const cut = failure as ExecutorError

  return { code: cut.code, retryable: cut.retryable, message: cut.message }
})

await new Promise((resolve) => setTimeout(resolve, 500))
await executor.dispose()

console.log(await outcome)

{
  code: 'CONNECTION_BROKEN',
  retryable: true,
  message: 'Worker 1 was shut down during the execution.'
}

wrapExecuteFailure reports a shutdown apart from a crash, so the code is CONNECTION_BROKEN and not WORKER_CRASHED, and retryable tells the caller the work can go to another replica. The grace buys the run no time of its own: the child is sent SIGTERM at once, and the grace bounds only what follows, the SIGKILL for a child that ignores it. A running execution takes its time from the server instead: shutdownDelay ahead of this call, then shutdownDrain inside it as the drainTimeout, and Stopping says how long a listener stays up under both.

No worker.crash is published for any of those deaths. Worker.shutdown clears the remote before it signals the child, and fail returns at once when the remote is already gone, so the exit event that follows announces nothing.

Events the pool publishes

pool.events is a typed EventEmitter<PoolEvents> and events.ts declares the payloads. Every event, every metric derived from one, and every span are tabulated under Telemetry.

Node’s default cap of ten would start warning under healthy load, because the listener count grows with concurrently retained executions: the bridge registers one handles.close listener per retained execution whose outbound duplicate it holds. So the pool lifts its own cap to unlimited.

instrumentation.ts is the subscriber the pool attaches to itself. Constructed with instrumented: false it gets silentInstrumentation instead, which builds no instruments and leaves the events published to nobody. Its barrel does not export attachInstrumentation, so a consumer cannot attach a second copy of what the pool owns, and shutdown detaches it before it clears the listeners. Instruments are built on the first event and not at attach time, because getMeter binds the global provider at the call while the tracer stays proxied, and a meter taken before the host registers an SDK stays a no-op for good.

Two measurements come from no event at all. measureDispatch wraps the caller’s own dispatch function and records executor.tool.duration around the round trip, and runInSpan wraps worker.execute in a pool.execute span with the execution id and the worker id, settled by the shared settleSpan. Without a registered SDK the OpenTelemetry API hands out no-op implementations, so both are safe and cheap in a process that measures nothing.

apps/daemon/src/daemon.ts counts the same events into the pool statistics its admin app reports, so codemode-executor status prints how many workers are up and how deep the queue is.

Layers above the pool

IsolateExecutor turns an ExecuteRequest into a PoolExecuteRequest. A defined request field wins over the construction-time default, an absent timeout falls back to DEFAULT_TIMEOUT, and a null timeout crosses as MAX_SUPPORTED_TIMEOUT, both of them the protocol’s own numbers. Grace and the isolate memory limit come from executor/defaults.ts: the two bounds the pool reads and never sets. On the way back it parses resultJson once through parseForDispatch, passes an error envelope and a grant through unparsed, and forwards invokeHandle and closeHandles to the pool unchanged. dispose() is pool.shutdown().

A daemon fills the same request from its own settings through buildServerOptions in apps/daemon/src/config.ts, which reaches six of those options and leaves every other one at the pool’s own number. Where a setting comes from names the six, and the operator guide sends a deployment to the reference for the numbers.

Where the numbers live

PoolOptions is all a host may set, and the published API lists every option beside its default. pool.ts declares them all. childExecArgv is the one it does not declare alone, because supervisor.ts defaults execArgv to the same ['--no-node-snapshot'] for a Worker built without a pool around it.

WorkerOptions declares the three supervision numbers, and the pool passes none of them, so every pooled worker runs on the value below and changing one is a change to this code. The grace is a parameter instead of a field, with a default a retirement takes and three call sites inside the pool that pass a number of their own.

Number	Value	Declared in
spawnTimeout, the allowance for a child’s first heartbeat	10000 ms	supervisor.ts
heartbeatInterval, the gap between health checks	5000 ms	supervisor.ts
heartbeatTimeout, the deadline on one health check	2000 ms	supervisor.ts
grace, the gap between the SIGTERM a shutdown sends a child and the SIGKILL that ends one that ignored it, which a crash and the spawn guard both pass as 0	1000 ms	supervisor.ts, exported as DEFAULT_SHUTDOWN_GRACE

Three numbers on a pool request belong to the caller and not to the pool: the timeout, the grace and the isolate memory limit. The layer that fills them in owns their defaults, and for a caller reaching the pool across a socket that layer is the server.