Why Faultbox
The first page of the story. If you read nothing else, read this and The verification model.
Every distributed system is a pile of assumptions
The moment your code talks to something it doesn’t control — a database, a queue, another service, a disk — it starts making assumptions:
- “The database is up.”
- “Writes succeed.”
- “The network is fast.”
- “This message is delivered exactly once.”
- “The clock moves forward.”
These assumptions are invisible. Nobody writes them down. They’re encoded
implicitly in the absence of error handling: the missing retry, the
missing timeout, the missing rollback, the err you logged and ignored.
The bugs that take systems down in production are almost never on the happy path. They live in the code that runs when an assumption breaks — code that, by definition, your happy-path tests never execute.
Why your existing tests can’t reach them
| You already have | What it proves | What it can’t do |
|---|---|---|
| Unit tests | A function computes the right answer | Nothing about service boundaries |
| Integration tests | The flow works against real dependencies | Can’t make a real Postgres refuse connections on cue |
| Load tests | Throughput ceilings, tail latency | Answers a QPS question, not a correctness-under-failure one |
| Production chaos | Runbooks, blast radius, human response | Slow feedback; the failure already shipped |
The gap is specific: provoking a precise failure, at a precise moment, and checking that the system still behaves correctly. That’s the gap Faultbox fills — between the integration test that can’t break the dependency and the production chaos drill that breaks it too late.
The question Faultbox answers
Not “does the system crash?” — that’s the easy case. The hard question:
When assumption X breaks, does the system still uphold property Y?
Does it return a useful error instead of hanging? Does it preserve data integrity instead of writing a half-row? Does it recover, or wedge itself open forever? Faultbox is built to ask that question thousands of times — every failure mode, against every critical flow, in every interesting ordering — and to give you an answer you can trust and reproduce.
How it works, in one breath
You write one spec describing your
system. Faultbox runs your real binaries (or containers), intercepts the
syscalls and protocol messages between them, and injects faults exactly
where you declared them — a denied write, a dropped Kafka message, a 503
rewritten over a 200. While the system runs under each fault, Faultbox
checks your invariants and temporal properties.
It explores the combinations
you couldn’t enumerate by hand, under a stated
determinism contract so the result is
reproducible, and hands you a report with a verdict.
Where it fits in your pipeline
Faultbox runs in pre-prod / local / CI and validates code paths. It composes with the tools around it rather than replacing them:
- You write a Faultbox spec while building the feature — it catches the missing retry, timeout, and circuit breaker before the PR lands.
- Staging runs production chaos against the deployed system — catching what Faultbox can’t: real LB/k8s failure modes, cross-region partitions.
- Production drills exercise the runbooks staging surfaced.
Skip step 1 and every code-path failure rides all the way to step 2, where feedback is measured in days and the chaos tooling becomes the bottleneck. Faultbox shifts that work left.
When not to reach for Faultbox
- Pure UI tests — Playwright / Cypress.
- Performance tuning — pprof, flamegraphs, a load tester.
- Contract testing — Pact does it better.
- In production — Faultbox uses
seccomp-notify; pointing it at a prod process is a footgun. Use real chaos tooling there.
Tagline: Faultbox finds the failure modes integration tests can’t reach, before production chaos has to.
Next: The verification model → — the single mental model every other page zooms into.