Benchmark

On the host clock a timeout bounds every run, V8 meters the isolate’s allocations against a memory limit, and the threat model names the boundaries around both. Both the suite and the harness live in apps/executor-benchmark, and the exact cases, sizes and bounds live next to the code they exercise. So this page states what stays true however those details move:

Family	What it asserts	Where it runs	What turns it red
Fairness, workload.spec.ts	Two measurements compare the same generated input and the same returned value.	every pnpm verify, and again at the start of every measurement	a changed seed, which fails the pinned digest, or a changed guest program, which fails the value diff
Termination, termination.spec.ts	The call settles at its bound whatever the guest is doing, and with the declared timeout text.	every pnpm verify	an interrupt delayed by seconds
Memory, memory.spec.ts	Every allocation path inside the isolate is metered, and the limit either ends the run or refuses the allocation.	every pnpm verify	a case metered against a limit larger than the one it was granted, or a reopened path V8 keeps outside the meter
Containment, containment.spec.ts	Every boundary of the threat model a program can reach has an attack written against it, and no attack escapes.	every pnpm verify	a boundary on that list with no attack written against it, or any of the four escapes below
Latency guard, latency.spec.ts	Cold start and steady state cost a bounded multiple of the host-compute baseline.	every pnpm verify	a delay planted in the worker’s stand-up, for the cold bound, or in the per-execution path, for the steady bound
Faults, nightly/faults.spec.ts	A worker killed by signal ends its own executions and no others, each as a typed fault a caller may retry, and the pool runs the next execution unchanged.	pnpm nx run executor-benchmark:nightly	a kill that ends an execution the worker was not running
Soak, nightly/soak.spec.ts	Results stay correct over thousands of executions, workers retire on their execution count, and host growth stays inside a fixed bound named by a constant beside the code.	pnpm nx run executor-benchmark:nightly	a worker retired by the memory bound instead of its execution count

Workload

WORKLOAD_SOURCE in apps/executor-benchmark/src/workload.ts is the workload, and its shape comes from the case corpus of the differential suite in apps/executor-testbed. It dispatches tools and awaits them together, builds small objects, parses URLs and matches a URLPattern, wires an AbortController into an EventTarget, round-trips a small FormData through a Response, and throws and catches one error. Byte-heavy primitives that corpus never touches are absent on purpose. Tool calls cross the dispatch to the host, the way a real integration runs them.

Input comes from a linear congruential generator with a fixed seed, and its SHA-256 digest pins it. A generator that drifted would let two measurements compare different inputs without saying so. Instead the fairness gate below turns red.

Fairness gate

One check compares the generated input against its pinned digest. Another diffs the guest’s whole returned value against an independent computation of the same workload on the host’s own natives, so a wrong field fails even where the length matches. The third runs the workload twice and compares what comes back. checkFairness is the first two as one function, and the measurement calls it before it prints a number. A red fairness gate voids every number the run produced.

Termination family

Time is measured on the host, not from inside the guest, so the outcome is bounded even for a guest that cannot be interrupted mid-step. Cases cover the shapes a guest gets stuck in: loops the interrupt catches at a back edge, the first of the four evil patterns OWASP lists, a nested quantifier whose backtracking never finishes on an input it cannot match, and a single V8 builtin the interrupt cannot cut. Each asserts that the call settles with the timeout text inside its allowance of timeout plus grace plus a fixed settlement margin. One journey through the pool asserts the same thing, and the worker runs the next execution afterwards.

Loop shapes and builtins end differently. For the loops and for the regular expression the guest work actually stops. But for the monolithic builtin only the outcome is bounded: the builtin runs to completion on the isolate’s thread, spending its own CPU until it returns and the isolate is disposed. Saturated CI machines are the reason for the settlement margin, and the failure class it guards against is measured in whole seconds.

Memory family

Paths that fill the heap in steps are cut by the engine, and the run reports the declared memory text. Paths that ask for one buffer the limit cannot grant are refused at the allocation site, with an error the guest can catch. Those cases count what the guest allocated before the refusal and bound the count near the limit. That bound ties the refusal to the isolate’s own limit and not to the machine.

The engine meters what passes through the allocator it was given. But V8 keeps some memory outside that allocator: a WebAssembly memory, and the growth of a resizable buffer, shared or not. The runtime closes each of those. One case per path asks for far more memory than the limit could grant, and asserts both halves: that the guest is refused instead, and that the process running the isolate did not grow while it asked. Without the second half, a path that returned something plausible and allocated anyway would read as closed.

Some allocation the guest causes happens on the host where the isolate’s meter cannot see it, console capture and a native outbound body among them. That work has bounds of its own, stated under Bounds around the engine. But one in-isolate edge is recorded, not asserted: a string that doubles itself outruns the collector and meets the engine’s own string-length ceiling before the limit does. That transient overshoot stays within the engine’s object ceilings, and the pool retires a worker whose memory stays past its bound.

Containment corpus

Attacks cover the empty global and the paths back to the host process, the module map as the only import, the bounds around the engine and the memory paths the meter cannot see, the tri-state outbound setting and the surfaces that could sit beside it, and the tool membrane and the completion envelope. They also cover what a run tries to leave behind for the next one: a global it wrote, a prototype it polluted, a timer it scheduled past its own end. One boundary sits outside the corpus altogether: the kernel policy of the manifests, which the kernel enforces around a deployed container and not around the process the corpus runs in.

Nothing the harness asserts rests on what the guest says about itself. Every program runs on one warm worker, so an attack meets the process its predecessor left behind. Four observations count as an escape:

• The worker did not survive the run.
• A value only the host process holds came back out.
• The execution after it did not start from a fresh global.
• The executor stopped serving.

Those four are written against classes of failure and not against the programs, so they catch a path nobody wrote a case for. Ending a worker by sending too much is one such path, and the first rule catches it. Beside them, each corpus entry pins what the run ended with and which tool calls and outbound requests the caller’s own functions were asked for. An undeclared provider or tool still arrives at the caller’s dispatch, because authority is enforced where the tool runs, and the tools guide shows that call arriving.

pnpm nx run executor-benchmark:probe -- <file>... puts a program that is in no corpus through the same four rules. It runs each file through the harness, prints one JSON report per file, and ends non-zero if any of them escaped. An agent writing attacks against this executor drives that write-run-read loop. --open gives the run an outbound function of the caller’s that records every request instead of performing it. Without the flag outbound is disabled, and most attacks want exactly that.

Nightly families

Both nightly families need more wall-clock time than pnpm verify can spend, and the nightly workflow runs them too. Real faults go into a working pool: a worker killed by signal while executions are flowing through it. Guest code that never ends and guest code that fills its heap run beside healthy work in the same rounds. Each settles inside its own envelope with a declared text. No worker dies.

Soak runs thousands of executions and reads what the pool publishes as they go by, and SOAK_EXECUTIONS sets how many. A worker hits its retireAfterExecutions count long before its memory bound, so every retirement the run records names executions as its reason.

Latency measurement

pnpm nx run executor-benchmark:measure reports three sections over the shipped IsolateExecutor, on wall-clock time, after the fairness gate passes. It prints its own run counts and pool shape, so the output states its own method.

Section	What the clock covers
Cold start	A fresh pool with one worker runs from construction to the first result, paying the child spawn, the runtime snapshot and the first execution.
Steady state	One warmed worker runs sequential executions with a fresh isolate for each, and the warm-up runs are discarded.
Throughput	The default pool runs a bounded number of executions in flight, and warm-up rounds start every worker before the clock starts.

Cold start and steady state also print their multiple of the host-compute baseline: the workload’s own computation on the host’s natives, and the ratio the guard below bounds. A number belongs to the machine and the toolchain that produced it, and the command prints both in its header. No number is worth quoting without them, so this page prints none. Anyone who wants one runs the command.

Latency guard

An absolute millisecond threshold would fail on the next machine. So each reading is divided by a batch of host computations run right next to it in time and sized to last about as long. Load spikes and slow machines move both sides of that ratio together. The guard bounds the median of the per-pair ratios.

Bounds live next to the assertions. A machine under load passes anyway, so a red bound says a path costs a multiple of what it should and not that the machine was busy. Running the measurement prints the multiple each path produced on the machine it ran on, and the room left under a bound is the distance from that multiple to it.

Reproducing in a pinned environment

Host toolchains vary, so the measurement also runs in a container built from the same base the daemon image uses. Resource caps take the place of weaker machine classes, and two CPUs and two gigabytes are one such class. One build and three runs follow, and the last of them runs the latency guard alone, straight through the Vitest binary, so that run needs no network:

docker build --file apps/executor-benchmark/Dockerfile --tag codemode-executor-benchmark .

# the measurement, on whatever the host gives the container
docker run --rm codemode-executor-benchmark

# the same, capped at two CPUs and two gigabytes
docker run --rm --cpus 2 --memory 2g codemode-executor-benchmark

# the latency guard alone
docker run --rm --cpus 2 --memory 2g --workdir /app/apps/executor-benchmark --entrypoint /app/node_modules/.bin/vitest codemode-executor-benchmark run src/latency.spec.ts

Containerized numbers are compared against containerized numbers. On a machine where Docker runs inside a virtual machine, macOS among them, they compare one run against another and never against bare metal.