Faultbox Documentation
Fault injection for distributed systems using Linux seccomp-notify. Write tests in Starlark that declare topology, inject faults, and assert on behavior.
Start with the concepts
The story, end to end: describe your system → generate a fault matrix → explore the state space deterministically → check invariants and temporal properties → read a verdict you can trust and reproduce.
Why Faultbox The gap between integration tests and production chaos — and why it matters The Verification Model One diagram every other page zooms into: spec → matrix → explore → check → verdict Invariants & Properties Safety, liveness, temporal — what an invariant is and how to find them Fault Matrix & Exploration Four injection layers, the scenario × fault grid, and state-space search Determinism & Reproducibility The L0–L5 ladder, the L1 contract, and replaying a failure Reading Reports & Verdicts PASS / FAIL / INCONCLUSIVE, who reads the report, and what they do
Getting started
Setup Install Faultbox, configure Lima VM for macOS, VS Code autocomplete Your First Fault Inject a write fault into a running binary in 5 minutes Your First Test Write a Starlark spec with services, healthchecks, and assertions
Reference
Spec Language Complete Starlark API: services, faults, assertions, protocols CLI Reference All commands and flags Error Codes Errno values for syscall fault injection Releases Per-version release notes and CHANGELOG Where Faultbox fits vs integration tests, load tests, and production chaos
Guides
Fault Testing Methodology Mental model, dependency matrix, the three layers, process from zero to covered Choosing Fault Levels When to use syscall vs protocol faults, decision table, combining both Invariants & Monitors Finding safety properties, encoding them as monitors, invariants under fault Spec Patterns Cookbook: API+DB+Cache, event-driven, microservice mesh
Tutorial roadmap
| Part | Chapters | Topics |
|---|---|---|
| 0: Prelude | 0. Setup | Environment setup, Lima VM, VS Code |
| 1: First Taste | 1. First Fault, 2. First Test | First fault, first test |
| 2: Syscall-Level | 3. Fault Injection, 4. Traces, 5. Concurrency, 6. Domains | Fault injection, traces, concurrency, domain-centric model |
| 3: Protocol-Level | 7. HTTP & Redis, 8. Databases | HTTP/Redis faults, database & broker faults |
| 4: Safety & Verification | 14. Invariants, 15. Monitors, 16. Partitions | Invariants & safety properties, monitors & temporal properties, network partitions |
| 5: Advanced | 9. Containers, 10. Scenarios, 11. Event Sources, 13. LLM & MCP, 23. Reports … | Containers, scenarios, event sources, mocks, bundles, LLM agents, reports |