Determinism & reproducibility

This is the foundation the whole verification model rests on. A failing test is only useful if you can reproduce it. If a “FAIL” can’t be reproduced, it’s a rumor, not a bug report — and a flaky test that fails once and passes on rerun trains the whole team to ignore it.

Faultbox treats reproducibility as a declared contract, not a hope. You state, at the top of the spec, how reproducible the run must be — and at strict levels the runtime fails the test the moment something escapes that contract, instead of silently poisoning the result.

determinism(level = "L1")            # the default contract: mediated-event determinism, strict
determinism(level = "L1", strict = False)   # detect leaks but only warn (forgiving local iteration)
determinism(level = "L0")            # plan determinism only

If you omit determinism() entirely, the spec runs at L1 with strict mode on — the safe default.

Why determinism is hard here

A spec runs real binaries. Real binaries do things Faultbox doesn’t control: read the wall clock, pull from /dev/urandom, resolve DNS, dial hosts nobody declared, spin up a background goroutine that phones home every 60 seconds. Every one of those is a source of non-determinism — a reason two runs of the “same” test might diverge. Each unmediated source is a future flake.

Faultbox’s job is to mediate as many of those sources as it can (it sits on the syscall and protocol boundary, so it sees them) and to detect and report the ones that leak past it.

The determinism ladder: L0 → L5

Determinism isn’t on/off — it’s a ladder. Each rung mediates more sources of non-determinism and makes a stronger reproducibility promise:

What ships in v0.13.0: L0 and L1. determinism() accepts only "L0" and "L1"; "L2"–"L5" parse but error at spec load — they’re the published roadmap, not yet runnable. The ladder is documented in full so you can write specs today that are ready for higher levels later.

The jump from L1 to L2+ is an engine change (a gVisor-class sandbox that mediates all I/O, not just the syscall/protocol boundary), which is why it’s roadmap rather than a flag.

L1, the contract you actually run against

L1 is where CI lives, so it’s worth understanding concretely. At L1 Faultbox guarantees reproducibility of the events it mediates (injected faults, intercepted syscalls, proxied protocol messages) and detects the I/O that leaks past it, emitting an unmediated_io event for each:

Strict mode (the default) fails the test at the first untolerated leak. This is the feature, not a nuisance: it converts “this test is flaky” from a Friday-afternoon investigation into a spec-load-time error pointing at the exact leak. When a leak is genuinely out of scope, you tolerate it explicitly:

# spec-wide — drift that affects every service
determinism(level = "L1", allow = ["clock"])

# per-service — a known-noisy dependency
service("telemetry", ..., nondeterministic_ok = ["network-unmediated"])

Tolerated categories still appear in the trace — tolerance suppresses the failure, not the visibility.

Known L1 limitations (v0.13.0)

Honest about the edges, so you don’t over-trust the green:

• VDSO clocks under-report. Go’s time.Now() and many libc clock calls use the kernel VDSO fast path, which bypasses seccomp-notify. The raw clock_gettime syscall is caught; the VDSO path is not.
• DoH/DoT looks like TCP/443. DNS tunnelled through HTTPS/TLS is classified network-unmediated, not dns.
• Interface matching is by port, not host. A connect() to any host on a declared port is treated as mediated.
• fs-unmediated isn’t emitted yet. File I/O outside declared volumes is not detected at L1.

Reproducing a failure: seed + replay

Every leaf in the exploration tree carries a seed. When a leaf fails, you replay that leaf:

faultbox test spec.star --seed 42       # re-run with the recorded interleaving
faultbox replay run.fb                  # replay a captured bundle

At L1 this reproduces the injected faults and mediated event ordering — not necessarily every SUT side-effect (that’s L3’s promise). This is why the L1 manifest pushes idempotent operations: idempotent retries survive any small replay drift.

Writing determinism-friendly specs

A few habits make the contract hold (the full per-level manifest lives in the reference):

• Declare every dependency as a service() + interface(). If your code talks to it and Faultbox doesn’t know, it’s a leak.
• Funnel I/O through seams — one injected HTTP client, one clock interface, one RNG — instead of scattered ad-hoc calls.
• Prefer eventually(p) over time.sleep(). Sleep is wall-clock and flaky at every level; eventually is a level-aware property check.
• Turn strict on in CI, off locally. determinism("L1", strict=True) in CI documents the promise and enforces it; omit or strict=False for forgiving local iteration.

→ Do it: Achieve strict determinism · Reference: Spec Language — determinism

Next: Reading reports & verdicts → — what Faultbox hands back, and who acts on it.