Build services that survive failure
Faultbox is fault injection for distributed systems. Most outages are ordinary failures your code never handled - Faultbox tests those paths: your service, failing dependencies, in CI.
v0.13.3 · Apache 2.0 · Linux (macOS via Lima) · single binary
The bugs you're shipping today
Most outages aren't exotic. They're ordinary failures - a refused connection, a slow reply, a crash between two writes - hitting code no test has ever run. Six classes cover almost all of them, and each is closed by a runnable spec.
fault(db, write=deny("EIO")) The database goes away - your API answers with a stack trace. Or worse, a 200. Missing timeout & runaway retry rules = [delay(path="/**", delay="800ms")] No deadline, retry without backoff - one slow dependency stalls every request in flight. Non-idempotent retry write=hold("charge") The first attempt succeeded. The retry charges the card twice. Partial failure deny after the first write The crash lands mid-transaction. Orphan rows survive the restart. Failed recovery eventually(consumer_caught_up) The broker comes back. Your consumer doesn't. Nobody notices until Monday. Bad responses, not outages rules = [response(status=200, body=garbage)] Downstream returns garbage with a 200 OK - and your parser takes the service down. The full catalog - what each class looks like in production, and the spec that closes it →
db = service("db", "./db", interface("main", "tcp", 5432),
healthcheck = tcp("localhost:5432"))
api = service("api", "./api", interface("public", "http", 8080),
env = {"DB_ADDR": db.main.addr}, depends_on = [db],
healthcheck = http("localhost:8080/health"))
def test_write_failure():
def scenario():
resp = api.post(path="/orders", body='{"item": "widget"}')
assert_eq(resp.status, 503, "degrade cleanly when DB writes fail")
fault(db, write=deny("EIO"), run=scenario)$ faultbox test faultbox.star
PASS test_write_failure (0.42s)
✓ fault(db, write=deny("EIO"))
✓ POST /orders → 503
✓ assert_eq(resp.status, 503)Every run is an artifact
A run doesn't end at PASS/FAIL in the terminal. Every run writes
a self-contained .fb bundle - spec, environment, full
event trace, replay script - and one command turns it into a
single-file HTML report: verdicts with drill-down, a swim-lane trace
per service, the fault plan and its coverage. The terminal tells you
that it broke; the report is where you understand
why. Attach it to the PR or the incident ticket.
Mix real and simulated dependencies
Run dependencies for real in Docker where that's cheap, simulate the rest with protocol-aware mocks - OpenAPI-generated if you have the contract. Faults work identically on both, so the spec doesn't care which is which.
api = service("api", image="myteam/api") # the one image you own
db = service("db", image="postgres:16-alpine") # real infra where it's cheap
auth = mock_service("auth", # simulated where it isn't
interface("main", "http", 8090),
routes = {"GET /token": json_response(200, {"ok": True})},
)
auth_down = fault_assumption("auth_down",
target = auth.main,
rules = [error(path="/**", status=503)],
)Mock services - HTTP, gRPC, Redis, Kafka, OpenAPI-generated → · Where Faultbox fits - vs integration tests, load tests, prod chaos →
Install
curl -fsSL https://faultbox.io/install.sh | shDetects your platform, downloads the latest release, verifies checksum. Or build from source.
Why Faultbox
Protocol-level injection
Inject faults at HTTP, HTTP/2, gRPC, Postgres, MySQL, Redis, Kafka, NATS, MongoDB, Cassandra, ClickHouse, AMQP, Memcached, TCP, and UDP protocol level. Target specific queries, paths, topics, or CQL statements via transparent proxy.
Mock services
Stand in for dependencies you can't run:
mock_service() HTTP/gRPC stubs,
redis.server(state=...), OpenAPI-generated responses.
In-process, no containers - and faultable exactly like real services.
Verdicts you can trust
Three-valued verdicts - PASS / FAIL / INCONCLUSIVE - with formal
finite-trace semantics. A timeout is never a green. Temporal
properties: eventually(), always(),
monitor() state machines over the event log.
Deterministic exploration
Name the unknowns with choose() and nondet();
Faultbox fans them out into a tree of leaves and walks every fault,
ordering, and choice. faultbox plan shows the tree;
seed-based replay reproduces any failing leaf.
Starlark specs
Topology, faults, and assertions in one .star file. No
YAML. No separate config language. The spec is executable code.
Syscall-level injection
Under the hood: deny, delay, or hold any syscall via seccomp-notify.
No eBPF, no ptrace, no code changes. Syscall families expand
automatically - write covers write,
writev, pwrite64.
Everything else - recipes, event sources, named operations, 15 protocols, LLM tooling →
How it works
.star file Powered by seccomp-notify - no ptrace, no eBPF, no code instrumentation. Faults are injected in the kernel, invisible to the target process.
Built for LLM agents
LLM agents write code. But who tests what happens when the database crashes, the network drops, or the disk fills up? Faultbox closes the loop.
Your LLM agent builds a microservice. It writes handlers, connects to Postgres, adds Redis caching.
One command from docker-compose. Every dependency gets fault scenarios - disk failures, network drops, slow queries.
faultbox init --from-compose JSON output with diagnostics: "write fault fired 3 times but service returned 200 - missing error handling in the persist path."
The agent reads the diagnostic, finds the code, adds error handling. Runs tests again. All pass. Commits with confidence.
faultbox init --claude creates slash commands and MCP config. Zero configuration.
Every LLM agent writing microservices needs to answer one question:
"What happens when things break?"
Faultbox is that answer.
LLM Integration Guide