Orphan Reaping¶
While a rightsize process is alive, normal lifecycle handles cleanup: stop(), JUnit's
@Sandboxed extension, the Backends.active() shutdown hook. The only
event that can leave a sandbox running with nothing left to stop it is the process itself dying
without running that cleanup — SIGKILL, an OOM-kill, a crashed CI step. Reaping exists purely
for that case, and is bound to the process, never to a test.
There are two layers, both driven by one ownership ledger: an always-on init-time sweep that finds a dead process's leftovers the next time any rightsize process starts, and an optional per-run watchdog that reaps within seconds of the crash instead of waiting for that next process. Neither is a background daemon — the watchdog is one short-lived process per run, spawned lazily and torn down with it.
The ownership ledger¶
Every process that starts a sandbox writes a small set of files under the rightsize cache
dir (the same directory backend-microsandbox provisions
into), keyed by that process's own run id:
runs/<run-id>.json— written atomically before the process's first sandbox: its PID, the process's own start time (used to defeat PID reuse — see below), and which backend it's using.runs/<run-id>.sandboxes— one sandbox name per line, appended before the backendcreatecall and removed after a successful stop. The file is therefore always a superset of what's actually live — a sandbox never exists without a line naming it.runs/<run-id>.networks— the same protocol for created networks.
This schema is identical byte-for-byte across rightsize's Kotlin, Rust, and TypeScript
libraries, because they all share one cache dir: a Kotlin process's sweep can judge — and reap
— a run left behind by the Node or Rust library just as readily as one of its own, as long as
that run's backend field names a backend the sweeping process is itself running (a docker
process never removes msb sandboxes, and vice versa; a cross-backend leftover simply waits for
a process on that backend to sweep it).
On a clean shutdown — the last container stopped, no networks left — the library deletes all three files as soon as that happens, not just at process exit; a later sandbox on the same process transparently recreates them. A crash leaves them in place, which is exactly the signal the sweep and watchdog act on.
Reuse sandboxes are structurally immune. A ContainerSpec with keepAlive = true (used by
container reuse) is never appended to .sandboxes at all — not
reaping a sandbox you deliberately want to survive the process is the whole point of reuse, so
it can't depend on some later opt-out; it's simply never listed. The same immunity extends to
every own-run cleanup path a backend already runs today, not just the ledger: the msb backend
never adds a keepAlive sandbox to the set its constructor shutdown hook and close() sweep,
and the docker backend labels a keepAlive container dev.rightsize.reuse=<12hex> at create
time instead of the run-id label, so its close()'s run-id label filter never matches it
either.
Liveness¶
A run is alive iff a process with the recorded PID exists and that process's own OS-reported start time matches the recorded one within two seconds. The start-time check is what defeats PID reuse: without it, a completely unrelated process that happens to land on a recycled PID would make a genuinely dead run look alive forever.
The init-time sweep¶
Runs exactly once per process, lazily, right after a backend resolves (the same once-per-process gate that already registers the shutdown hook). For every other run's record under the cache dir:
- Skip the sweeping process's own run.
- An unparseable record younger than an hour is left alone (it might just be mid-write by its still-alive owner); older than an hour, it's presumed abandoned and cleaned up.
- A run whose backend doesn't match the sweeping process's own backend is left for a process on that backend.
- A run that's still alive is left completely untouched.
- A dead run is reaped: every name in its
.sandboxesis removed via the active backend'sremoveByName(best-effort — "not found" is silently ignored, since two processes racing to sweep the same dead run is expected and harmless), every network in.networksis removed the same way, then all three of that run's files are deleted.
The watchdog¶
Spawned once per process, lazily, right before the process's first sandbox create (once the run
record already exists on disk). A small script — POSIX sh on macOS/Linux, PowerShell on
Windows — written once under <cache-dir>/reaper/ and reused by every run (its content is
generic, parameterized entirely by the arguments a given run passes it), spawned detached with
its stdin connected to a pipe whose write end this process holds and never explicitly closes.
All other stdio is redirected to null, so CI runners never hang on an inherited handle.
That pipe is the entire mechanism: as long as the process is alive, the write end stays open and
the script blocks reading stdin. The moment the process exits — cleanly or via SIGKILL — the
OS closes that file descriptor, the script's read hits EOF, and it:
- Kills every name in
runs/<run-id>.sandboxesvia the backend CLI (an msb run getsstopthenrm; docker gets onedocker rm -f), retrying once if the output carries msb'serror: database error:signature — the same transient-migration-race classifier the boot path itself retries on. - Removes every network in
runs/<run-id>.networks. - Deletes the run's three ledger files.
An empty or missing .sandboxes (the ordinary clean-shutdown case, if the watchdog somehow
outlives the process's own cleanup) just deletes the record files — reaping on EOF is
idempotent, so there's no separate "was this a crash" signal to compute.
The docker-remote caveat. A watchdog is a local process: it cannot outlive the machine it runs on. If a CI runner is torn down mid-run against a remote docker daemon, the watchdog dies with it before it can fire, and the containers it would have reaped keep running on that daemon. The next rightsize process to talk to that same daemon still finds and sweeps them — the init-time sweep doesn't care where its previous run happened, only that the record and the sandbox are both still there — so nothing leaks permanently, it just isn't reaped within seconds the way a local-daemon crash is.
The RIGHTSIZE_REAPER switch¶
| Value | Sweep | Watchdog |
|---|---|---|
on (default) |
yes | yes |
sweep |
yes | no |
off |
no | no |
Unknown values are treated as on. Set RIGHTSIZE_REAPER=sweep to skip spawning the watchdog
process entirely (e.g. a sandboxed CI environment that can't spawn detached children) while
still getting next-run cleanup; set RIGHTSIZE_REAPER=off to disable reaping altogether.
What this doesn't cover¶
Orphan reaping only reacts to process death. If your own code forgets to call stop() while
the process stays alive, nothing here helps — that's what @Sandboxed/@Container, try/
finally, and the Backends.active() shutdown hook are for. See How It
Works for the full container lifecycle.