Failures, and what to do with each

Failures divide once, in one place, and the executor holds that line everywhere. Program failures are data. One arrives inside the envelope as error, and from your host’s point of view the call succeeded: the model wrote something that did not work, an ordinary event and one the model can fix. Executor failures are exceptions. execute rejects with one, and each carries a code and a retryable flag. Your host tells the two apart in a single branch:

const outcome = await executor.execute({ code }) // throws only for infrastructure

if (outcome.error !== undefined) {
  // the program failed; show this text to the model and let it try again
}

outcome.error carries every way a program can fail: a syntax error, a throw, a rejected promise, a bound the executor enforced, and a refusal the program could have caught and did not. Nothing a program writes makes execute reject.

Reading a fault

Codes cross a process boundary and an RPC boundary. But class identity does not: it dies in serialization, so branching on instanceof works in one arrangement and quietly stops working in the other. The pool speaks the same RPC protocol over its IPC channel that the remote executor speaks over a WebSocket. So this holds in your own process as well as over a network.

import { ExecutorError, rehydrateExecutorError } from '@supolka/codemode-executor'

try {
  return await executor.execute({ code })
} catch (failure) {
  const fault = rehydrateExecutorError(failure)

  if (fault instanceof ExecutorError && fault.retryable) {
    await wait(backoff)

    return await executor.execute({ code })
  }

  throw fault
}

rehydrateExecutorError rebuilds the typed error in your own process from the code the wire carried, and passes through untouched anything whose code it does not recognize. Provider rejections and throws from your own code keep their shape. Do not drop the wait. A queue that was full the instant it rejected is still full an instant later. Every fault carries retryable, and that flag is the executor’s own answer to whether trying again can work. Read it instead of keeping a table of which codes deserve a second attempt.

Fault codes, and what each one asks of you

The code answers a different question from the flag. It says what broke, and what broke decides where the fix goes. Read the flag to decide whether to try again, and the code to decide what to change:

Code	What it means	What to do
QUEUE_OVERFLOW	Every worker is busy and the waiting line is at its bound.	Back off and retry. Scale on the queue depth.
WORKER_CRASHED	The child process under the execution died or never started.	Retry once. The pool has already replaced the worker.
CONNECTION_BROKEN	The transport under a call died, or the executor is disposed.	Read retryable: a broken session reopens on the next call, a disposed executor does not.
CONNECTION_REJECTED	connect failed on the socket or on the handshake.	Check the address and whatever admits the connection.
PROTOCOL_MISMATCH	The two sides disagree on the protocol version.	Deploy matching versions. Reconnecting never heals it.
RESPONSE_TOO_LARGE	The server measured the answer above its response limit.	Return less, or raise the limit on the server.
HANDLES_CLOSED	The execution named by invokeHandle or closeHandles holds no handles.	Execute again with retainHandles for a fresh executionId.
INTERNAL	An invariant of the executor broke, or the pool is shutting down.	Report it. No change to the request makes it go away.

A host meets two of them in production, and both are always retryable:

ExecutorError | QUEUE_OVERFLOW  | retryable true | The queue is at its bound of 1. Retry when a running execution settles.
ExecutorError | WORKER_CRASHED  | retryable true | Worker 1 failed to start.

Two more are worth recognizing on sight, because they mean a wiring problem and not a load problem. Connecting to an address with no listener gives CONNECTION_REJECTED | retryable false | Executor handshake failed for ws://127.0.0.1:9. Calling a disposed executor gives INTERNAL | The pool is shutting down. in your own process and CONNECTION_BROKEN | RemoteExecutor is closed over the wire. The failure reference enumerates every code beside the file it is raised in, along with the fault classes that carry a text written to the same grammar as the declared failures below.

Texts a model reads

Declared failures are written for the model and not for a log. Each one names what happened in its first sentence and says what remains possible in the next. That second sentence turns a failed run into a better next attempt and not a loop:

Execution ran out of memory at its 32 MB limit. Work through the data in smaller pieces and keep only what you return.

Show that sentence to the model. It was written to be shown, and the 32 MB in it is the limit the execution actually ran under. Every size in one of these texts is grouped for one locale, so the sentence reads the same wherever the executor runs. The failure reference quotes each of them beside the class that composes it. Where a text lands depends on which boundary reported it. A bound the executor enforced, and anything that failed before the program ran, arrive in the envelope’s error. Something the program itself touched arrives as a rejection it can catch, so the model’s own try can work around it:

Where it lands	What the program catches
The envelope’s error	A timeout, the memory limit, a provider name, an unmapped import
A TypeError from fetch	Outbound disabled, and an outbound request past the payload bound
A DOMException named NotSupportedError	A declared absence such as caches
A plain Error at a tool call site	The not-found text, the completed text, a tool call past the payload bound

Missing capabilities and what they answer

Names this runtime does not carry are usually absent completely: reading one answers undefined, and a program that checks for a feature takes the branch it already has. But a few names exist and reject at their first touch instead, each with a sentence saying why. A bare undefined there would send the model hunting for a bug that does not exist.

await executor.execute({
  code: `async () => {
    try {
      await caches.open('x')
    } catch (failure) {
      return { name: failure.name, message: failure.message }
    }
  }`,
})

{
  result: {
    name: 'NotSupportedError',
    message: 'Cache storage is absent from this runtime because nothing survives an execution. Return what should persist in the result and take it back as an input on the next call.'
  },
  logs: []
}

caches is the entry point to the Cache API, and here it exists for the sole purpose of explaining its own absence. CompressionStream, DecompressionStream and the two buffer growth operations answer the same way, each with its own reason. Runtime deviations lists every one.

Failures your host causes

Four failures point at your code and not at the program’s.

The dispatch answered outside the result envelope. means your dispatch answered with something that is not an object at all. Objects of another shape get through. The proxy reads result off one, so the call answers undefined and no failure is reported. Use createEnvelopeDispatch and neither happens. The fetch handle answered with a value that is not a Response. and The fetch handle answered outside the response wire shape. mean a retained handler returned the wrong thing, and invoking a handle says what a handler has to answer. Provider name failures land before a single line of the program runs, so a host generating names from user data validates them first, or reads the text and renames.

Failures a program cannot cause

User-code failures never throw on your side, and no program can make your host’s execute reject by writing bad code. If one does, that is a defect in this executor and not in the program. Unhandled rejections inside the isolate do not replace the result a program already reported:

await executor.execute({
  code: `async () => { Promise.reject(new Error('ignored')); return 'kept' }`,
})
// { result: 'kept', logs: [] }

A tool call that arrives after the run settled does not reach your data. The sandbox answers it with Execution has already completed. without entering your dispatch, and the state latch the tools guide shows closes that same path a second time. A program cannot end the worker holding it by answering with too much. Each of those is a boundary with a corpus attacking it on every gate, described in the threat model.

Next

This page divides failures into the two kinds a host handles. Modules ships a program what it needs to import. Handles lets one keep serving after it returns.