One API, three sandboxes

Every agent platform ends up with the same component: a service that takes a command from a model, runs it somewhere the model cannot hurt anything, and returns the output. The interesting question is what "somewhere" is. A container shares the host kernel. gVisor puts a second, user-space kernel between the process and the real one. Firecracker gives the process a kernel of its own inside a microVM. Each answer costs something different in startup time, syscall overhead and memory, and the only honest way to choose is to run the same work on all three.

cordon is that service, built so the comparison is a table rather than an opinion. One gRPC API for agents (Acquire, Exec, Release, or Run for one-shot calls), one FastAPI control plane for operators, and behind them a five-method runtime contract — create, exec, destroy, snapshot, restore — that runc, gVisor and Firecracker each implement. runc and gVisor share one OCI bundle and differ only in which binary receives it. Firecracker boots a VM over its API socket and talks to a small agent inside it over vsock.

The suite has to be able to say no

The escape suite is thirteen programs: reach the network, open a raw socket, write outside the sandbox, read /etc/shadow, allocate a gigabyte under a 128 MB cap, burn CPU, sleep forever, fork five hundred times, run three children that each stay under the per-process cap but not together, flood stdout, write a 200 MB file. Each runs in a fresh sandbox, and each has a predicate for what it wanted — a fork bomb that was refused by the pids cgroup exited 1, but so would a fork bomb that ran out of memory on the host, and the table has to tell those apart.

There is also a fourth runtime, host, which is not a sandbox at all: rlimits and nothing else, so the whole stack runs in CI without privileges. Against it the suite reports five of thirteen escaped — the socket connects, /etc is writable, /etc/shadow reads, mount succeeds, and three children under the per-process cap happily exceed it together. Against runc: thirteen of thirteen contained, each row naming the control that did it — network, filesystem, memory, processes, cpu, file_size, wall_clock. A suite that cannot fail a weak runtime cannot be trusted to pass a strong one.

Naming the refusal

The result of every command says which control stopped it, and that comes from what the kernel counted, not from stderr. An out-of-memory kill shows up as an increment in the cgroup's memory.events; a pids-limit hit in pids.events; CPU and file-size limits arrive as SIGXCPU and SIGXFSZ. The counters are cumulative for the container's life, so each exec reads them before and after and reports the difference — the first version read them once and, after a single OOM, called every later command in that sandbox a memory failure.

A restore instead of a boot

The number every sandbox platform fights is time to shell: from "give me a sandbox" to "a command ran in it". cordon keeps a warm pool per runtime; Acquire takes a ready sandbox and a filler thread starts its replacement. For Firecracker the pool boots one golden VM, snapshots it, and fills itself by restoring the snapshot — the per-sandbox cost is a memory file load, not a kernel boot. On runc, on the machine this was developed on, a cold start (create plus one exec) is 70 ms at p50; from the pool it is 22 ms, which is almost entirely the exec itself. Twenty idle containers cost 2.5 MB each.

Three things the tests found

A detached container holds the pipes. runc run --detach returns, but the container's init inherits whatever stdout and stderr it was given and keeps them open for its whole life. Call it with a pipe and wait for EOF and you wait forever. The fix is a file, not a pipe, for anything that will outlive the call.

Forking inside a gRPC thread aborts. The host runtime set its rlimits in a preexec_fn, which forces Python's subprocess down the plain fork() path instead of vfork. fork() runs every registered atfork handler, and gRPC's handler aborts a child forked from one of its own threads. The limits are now applied from the parent with prlimit after spawn, which needs no fork at all.

Binding the host's /etc read-only still exposes /etc/shadow. Read-only means unwritable, not unreadable. In the development mode that borrows the host's directories, the secret files are masked with /dev/null; with a real root filesystem the file is the image's, not the host's.

What comes next

The three-way comparison needs a machine with KVM, which a laptop does not have and most cloud VMs do not expose. The repository's Terraform brings up a GCE instance with nested virtualization and a cloud-init that installs runc, runsc, Firecracker, a guest kernel and a guest root filesystem built from the same image for all three runtimes. Part two is the table that comes out of it: time to shell cold, pooled and restored; syscall-heavy and CPU-bound work relative to the bare host; memory per sandbox at fifty concurrent.