The benchmark

The executor promises two things about a run: a timeout that always lands, and a memory limit that meters what the isolate allocates. It also names a set of boundaries in the threat model. None of that is taken on trust. Three proofs run on every pnpm verify: a property suite proves the two bounds, an adversarial corpus attacks each of those boundaries, and a guard bounds the latency of the real workload against the machine it happens to be running on. But two families need more time than a gate can spend, so they run nightly. Measuring the latency of the shipped executor takes one command anyone can rerun.

This page is the method: what runs, what is asserted, and how to obtain every number yourself. Both the suite and the harness live in apps/executor-benchmark. Exact cases, sizes and bounds live next to the code they exercise, so this page states what holds however those details move.

Workload

WORKLOAD_SOURCE in apps/executor-benchmark/src/workload.ts is the workload, and its shape was read off the case corpus of the differential suite in apps/executor-testbed, not invented. It dispatches tools and awaits them together, builds small objects, parses URLs and matches a URLPattern, wires an AbortController into an EventTarget, round-trips a small FormData through a Response, and throws and catches one error. Byte-heavy primitives that corpus never touches are absent on purpose. Tool calls cross the dispatch to the host, the way a real integration serves them.

Input comes from a linear congruential generator with a fixed seed, and its SHA-256 digest pins it. A generator that drifted would let two measurements compare different inputs without saying so. Instead the gate below turns red.

Fairness gate

Every number is valid only while this gate is green. Two checks make it up. One holds the generated input to its pinned digest. The other diffs the guest’s whole returned value against an independent computation of the same workload on the host’s own natives. A wrong field therefore fails the diff even where the length matches. Both run as a spec on every pnpm verify and again at the start of every measurement, and a red gate voids the run. Two mutations prove the gate fails when it should: a changed seed fails the digest, and a changed guest program fails the value diff.

Termination family

Here the property has two halves: a run settles at its bound whatever the guest is doing, and it settles with the declared timeout text. Time is measured on the host, not from inside the guest, so the outcome is bounded even for a guest that cannot be interrupted mid-step. Cases cover the shapes a guest gets stuck in: loops the interrupt reaches at a back edge, the first of the four evil patterns OWASP lists, a nested quantifier whose backtracking never finishes on an input it cannot match, and a single V8 builtin the interrupt cannot cut. Each asserts that the run settles with the timeout text inside its allowance of timeout plus grace plus a fixed settlement margin. One journey through the pool asserts the same thing, with the worker still serving afterwards.

Loop shapes and the builtin shape end differently, and the difference is the honest boundary of the property. For the loops and for the regular expression the guest work actually stops. But for the monolithic builtin only the outcome is bounded: the builtin runs to completion on the isolate’s thread, spending its own CPU until it returns and the isolate is disposed. Saturated CI machines are the reason for the settlement margin, and the failure class it guards against is measured in whole seconds. So a mutant that delays the interrupt by seconds turns the family red.

Memory family

One property covers both endings: an allocation path inside the isolate is metered, and the limit ends it. Each case pins one of the two bounded endings that follow. Paths that fill the heap in steps are ended by the engine, and the run reports the declared memory text. Paths that ask for one buffer the limit cannot grant are denied at the allocation site, with an error the guest can catch. Those cases count what the guest held before the denial and bound the count near the limit. That bound proves the denial came from the isolate’s limit and not from the machine, and a mutant that meters a larger limit turns the denial cases red.

The engine meters what passes through the allocator it was given. But V8 keeps some memory outside that allocator: a WebAssembly memory, and the growth of a resizable buffer, shared or not. The runtime closes each of those. A case per path asks for far more memory than the limit could grant and asserts both halves: that the guest is answered instead, and that the process running the isolate did not grow while it asked. Without the second half, a path that answered something plausible and allocated anyway would read as closed. Mutants that reopen one path prove it.

Some allocation the guest causes lands on the host where the isolate’s meter cannot see it, console capture and a native outbound body among them. That work carries bounds of its own, stated under Bounds around the engine. But one in-isolate edge is recorded, not asserted: a string that doubles itself outruns the collector and meets the engine’s own string-length ceiling before the limit does. That transient overshoot stays within the engine’s object ceilings, and the pool’s worker memory bound retires a process still holding it.

Containment corpus

Every boundary of the threat model a program inside the isolate can reach is attacked by a program written to break it, and the corpus runs on every pnpm verify. Those programs cover the empty global and the paths back to the host process, the module map as the only import, the bounds around the engine and the memory paths the meter cannot see, the tri-state outbound setting and the surfaces that could stand beside it, and the tool membrane and the completion envelope. They also cover what a run tries to leave behind for the next one: a global it wrote, a prototype it polluted, a timer it scheduled past its own end. A spec beside the corpus turns red when any boundary on that list has no program written against it. One boundary the corpus does not reach at all is the kernel policy of the manifests, which stands outside the process the corpus runs in.

Verdicts belong to the host and never to the program, and that difference makes the corpus a gate and not a demonstration. Every program runs on one warm worker, so an attack meets the process its predecessor left behind. A run has escaped when it ends its worker, when a value only the host process holds comes back out, when the execution after it starts from a global that is not the one a fresh isolate carries, or when the executor stops serving afterwards. Those rules are written against classes of failure, not against the programs. So they catch a path nobody wrote a case for: every way a guest could once end its worker by sending too much was caught by the first rule before anyone thought to classify it.

Beside the rules, each corpus entry pins what the run ended with and which tool calls and outbound requests the caller’s own functions were asked for. One of those endings is worth reading twice: an undeclared provider or tool still reaches the caller’s dispatch, because authority is enforced where the tool runs. And nothing the corpus asserts rests on what the guest says about itself.

The same judgment is available from the command line, for a program that is in no corpus at all. pnpm nx run executor-benchmark:probe -- <file>... runs each file through the harness, prints one JSON report per file, and ends non-zero if any of them escaped. An agent writing attacks against this executor drives that loop. Adding --open hands the run an outbound function of the caller’s that records every request instead of performing it. But without the flag outbound is disabled, and most attacks want to meet exactly that.

Nightly families

Two properties need more time than a gate can spend, so they run from pnpm nx run executor-benchmark:nightly and on the nightly workflow.

The fault family puts real faults into a working pool. A worker is killed by signal while executions are flowing through it, and three things then have to hold. That kill takes the executions the worker was holding and no others, each of them ends as the typed fault a caller may retry, and the pool serves the next caller unchanged. Guest code that never ends and guest code that fills its heap run beside healthy work in the same rounds. Each settles inside its own envelope with a declared text, and no worker dies.

The soak runs thousands of executions and reads what the pool publishes as they go by. Three properties hold across the run. Every result is the right one. Every worker retires on its execution count and not on the memory bound behind it: the churn bound keeps a worker below that memory bound. And the growth of the host process stays inside a fixed bound the constant beside the code names, however many executions ran. SOAK_EXECUTIONS sets the count.

Latency measurement

pnpm nx run executor-benchmark:measure reports three sections over the shipped IsolateExecutor, on the wall clock, after the fairness gate passes. The command prints its own run counts and pool shape, so its output carries its method with it.

Cold start is a fresh pool with one worker, from construction to the first result, which pays the child spawn, the runtime snapshot and the first execution. Steady state is one warmed worker running sequential executions with a fresh isolate for each, warm-up runs discarded. Throughput is the default pool with a bounded number of executions in flight, every worker started by warm-up rounds before the clock starts.

Cold and steady lines also carry their multiple of the host-compute baseline: the workload’s own computation on the host’s natives, and the relation the guard below bounds. A number travels with the machine and the toolchain that produced it, and the command prints both in its header. No number is worth quoting without them, so this page carries none. Anyone who wants one runs the command.

Latency guard

Cold start and steady state are the two latency promises a consumer builds on, so a spec beside the property families bounds them on every pnpm verify. An absolute millisecond threshold would fail on the next machine. So each reading is bounded as a multiple of the host-compute baseline instead. Every reading is divided by a batch of host computations run right next to it in time and sized to last about as long. So a load spike or a slow machine moves both sides of that ratio together. The guard bounds the median of the per-pair ratios.

Bounds live next to the assertions. A machine under load passes anyway, so a red bound says that a path costs a multiple of what it should, and not that the machine was busy. The measure command prints the multiple each path produced on the machine it ran on. Room left under a bound is the distance from that multiple to it. Two planted slowdowns prove the guard can fail: a delay in the worker’s stand-up turns the cold bound red, and a delay in the per-execution path turns the steady bound red.

Reproducing in a pinned environment

The host toolchain is a variable, so the measurement also runs in a container built from the same base the daemon image uses:

docker build --file apps/executor-benchmark/Dockerfile --tag codemode-executor-benchmark .
docker run --rm codemode-executor-benchmark

Resource caps stand in for weaker machine classes. Two CPUs and two gigabytes are one such class: docker run --rm --cpus 2 --memory 2g codemode-executor-benchmark. Containerized numbers are compared against containerized numbers. On a machine where Docker runs inside a virtual machine, macOS among them, they answer relative questions and never bare-metal ones.

The latency guard runs in the same image under the same caps, straight through the Vitest binary so the run needs no network:

docker run --rm --cpus 2 --memory 2g --workdir /app/apps/executor-benchmark --entrypoint /app/node_modules/.bin/vitest codemode-executor-benchmark run src/latency.spec.ts