Features
v0.13.3 · Apache 2.0 · Linux (macOS via Lima)
Protocol-Level Fault Injection
Inject faults at the protocol level via transparent proxy. Target specific HTTP paths, SQL queries, Redis commands, or Kafka topics - without touching the network stack.
fault(service) = syscall level. fault(service.interface) = protocol
level. Same builtin, different dispatch.
Return HTTP 503 for POST /orders, inject Postgres query errors,
drop Kafka messages on specific topics.
Mock Services - Simulate What You Can't Run
The dependencies you don't own - another team's API, a paid SaaS, an auth provider - become in-process, protocol-aware stand-ins. No containers, no images, first test in under 100 ms. And they accept the same fault rules as real services.
mock_service() for HTTP, HTTP/2, gRPC, TCP, UDP;
redis.server(state=...), kafka.broker(topics=...),
mongo.server(collections=...) for infrastructure.
http.server(openapi="petstore.yaml") - response stubs
generated from the OpenAPI document; typed gRPC mocks from your protos.
The proxy layer rewrites mock responses with the same
error(), response(), delay() rules -
one fault vocabulary across mocks and real dependencies.
Reports - Where the Analysis Happens
Every run writes a self-contained .fb bundle: spec,
environment, full event trace, replay script. One command -
faultbox report run.fb - turns it into a single HTML
file that opens anywhere: swim-lane trace per service, per-test
verdicts with drill-down, the plan tree and its coverage. The
terminal says PASS or FAIL; the report is where you understand
why.
CSS, JS, and compressed trace data inlined - no server, no network. Attach it to a PR, an incident ticket, or an email.
Every service a lane, every intercepted operation an event - fault decisions and causality visible at a glance.
faultbox replay run.fb re-executes the exact
fault schedule that produced the report.
Deterministic Exploration
Name the unknowns in your spec with choose() and nondet(),
and Faultbox expands them into a tree of test executions - one leaf per
fault, ordering, and choice. faultbox plan shows the tree before
you run it; every failing leaf replays from its seed.
choose("backend", [primary, replica]) and
parallel(..., interleavings=...) fan out into multiple leaves,
each with a stable LeafID in the report.
faultbox plan --coverage analyses the tree statically - no
services launched. --check-cost --max-instances N guards blow-ups.
Every leaf has a seed. Failed? Replay with --seed 42 for an
identical interleaving - deterministic debugging.
Starlark Specs
Topology, faults, and assertions in one .star file.
Starlark is a Python dialect - if you know Python, you know Starlark.
No YAML, no separate config language. The spec is executable code.
service(), interface(), depends_on,
healthcheck - declare topology as code.
assert_eq(), assert_eventually(),
assert_never(), assert_before() - value checks
and temporal properties on the syscall trace.
Register happy paths with scenario(), then
faultbox generate creates failure tests automatically.
Event Log & Traces
Every intercepted syscall is recorded with vector clocks, service attribution, and file paths. Assert on internal behavior, not just inputs and outputs.
"The WAL write happened before the response" -
assert_before() proves ordering guarantees.
--shiviz trace.shiviz produces a space-time diagram
with causal arrows between services.
Capture before/after a refactor. faultbox diff shows
exactly what behavioral changes you introduced.
Binary & Container Modes
Run local binaries for fast development, or real infrastructure in Docker containers for integration testing. Same spec, same assertions, same faults.
binary="./my-service" - fork+exec with seccomp filter.
Fastest iteration, no Docker needed.
image="postgres:16" - Docker containers with faultbox-shim
entrypoint. Test against real Postgres, Redis, Kafka.
Monitors & Network Partitions
Define safety invariants as monitors that run on every syscall event. Simulate network partitions between specific services.
Callbacks that fire on every matching event - fail immediately if an invariant is violated.
partition(orders, inventory, run=scenario) - bidirectional
network split, other connectivity intact.
Named Operations
Group related syscalls into logical operations. Fault "persist" instead of "write + fsync". Path filters target specific files.
ops={"persist": op(syscalls=["write","fsync"], path="*.wal")} -
fault the WAL persist operation, not individual syscalls.
Recipe Library
Curated, protocol-specific failure wrappers ship embedded in the
faultbox binary. Load them via the @faultbox/
prefix - no filesystem setup, no network fetch. Each recipe encodes a
canonical error message, status code, or incident pattern drawn from
real postmortems.
load("@faultbox/recipes/mongodb.star", "mongodb") -
works from any project, no recipes/ directory needed.
Ships with every binary.
mongodb.disk_full() and postgres.disk_full()
coexist. One import per protocol, zero name collisions.
Say cassandra.unavailable() instead of remembering
"Cannot achieve consistency level QUORUM". Recipes stay in sync
with real driver behavior.
faultbox recipes list and recipes show <name>
- browse the catalog without reading source.
Under the Hood: Syscall-Level Fault Injection
The engine beneath it all: intercept any syscall via Linux's seccomp-notify and decide - allow, deny, or delay. No eBPF, no ptrace, no code changes. Works on any binary: Go, Rust, Java, Python, C.
write automatically covers write, writev,
pwrite64. Think in operations, not syscall numbers.
Fault only writes to /data/*.wal - stdout, TCP, and other writes
are unaffected. Powered by fd→path resolution via /proc.
deny("EIO", probability="30%") or deny("EIO", trigger="after=5") -
intermittent failures and trigger-on-Nth-call.
LLM-First Design
New in v0.2.0
Faultbox is designed for both human engineers and LLM agents. Structured JSON output, MCP server, Claude Code integration - everything an agent needs for an autonomous code → test → fix loop.
faultbox mcp - 6 tools for Claude, Cursor, and any MCP client.
Run tests, generate specs, analyze failures natively.
--format json - machine-parseable results with fault info,
syscall summary, and actionable diagnostics.
faultbox init --claude - slash commands (/fault-test,
/fault-generate, /fault-diagnose) and auto-MCP config.
faultbox init --from-compose - zero-effort spec generation.
Detects protocols, wires dependencies, generates happy-path tests.
Not just "test failed" - structured hints like "write fault fired but service returned 200 - missing error handling in persist path."
ghcr.io/faultbox/faultbox image + GitHub Action for
automated fault testing on every PR.
Recent Releases
| Version | Highlights |
|---|---|
| v0.13.3 current | Bug fixes + stricter spec loading from the July 2026 doc audit. Five silent no-ops fixed: drop(query=/command=) was meant to scope a drop to a SQL statement or Redis command but silently dropped all traffic (an empty pattern matched everything) - now it matches; Kafka duplicate(topic=) was a dead rule - now it re-sends the produce so the consumer sees the message twice while the producer still gets one ack; proxy events dropped the action/protocol fields, so assert_eventually(...action == "error") could never fire - now they carry them. Spec loading is stricter: deny() rejects errno names outside the supported table (widened with the documented ENOSYS/EDEADLK/ELOOP/EDQUOT/ENOLCK), and service() plus every proxy fault builtin (response/error/drop/delay/duplicate) reject unknown keyword arguments with migration hints instead of silently ignoring them - specs relying on silently-dropped kwargs (cmd=, http=, tcp=, name=) must switch to the documented forms. subject= is now a topic= alias for NATS matchers; proxy_conn_close/proxy_stall carry the protocol field. Full host suite + go test -race + go vet green; Lima sweep 21/21 PASS. |
| v0.13.2 | Finite-trace verdict semantics (RFC-049). Temporal verdicts are now grounded in finite-trace logic. A test() that ends by hitting its timeout is always INCONCLUSIVE, never PASS - even if every eventually() it declared was satisfied before the deadline - because the body never reached a declared completion (natural return or terminate_when=), so the run is a truncated prefix and a green verdict would over-claim. An unbounded always(p) (no between=) under timeout is INCONCLUSIVE for the same reason; at natural completion or terminate_when= it stays a definitive PASS. New vacuous_property warning event: when an always(p, between=) start anchor never fires, the window never opens and the predicate is never evaluated - the verdict stays PASS (the window may be legitimately untriggered) but the warning surfaces a typo'd anchor instead of hiding as a silent green. Ships alongside the documentation and site restructure around the six bug classes. Full host suite + go vet green; Lima sweep 21/21 PASS across the 6 integration spec suites. |
| v0.13.1 | First field-eval fixes (v0.13.0 → truck-api). --test matching no tests now exits non-zero - a typo no longer reads as a green suite in CI - and a collapsed fault_matrix name (test_matrix_create_order) selects every expanded cell under it. Unstartable targets fail fast with the named path (exec <path>: no such file or directory) instead of a 60-second healthcheck timeout with a misleading exit_code=0 - both the binary-mode launch and the container shim carry the path, and the healthcheck races against session exit. step_recv trace events record the response body (2 KB) on non-2xx HTTP responses, so a 400/500 reads straight off the trace or report. New make install-lima cross-compiles and installs both faultbox and faultbox-shim into the Lima VM (container mode needs both side by side); a stale hardcoded shim path and an incorrect VM_PROJECT were cleaned up. The monitor() signature change from v0.13.0 keeps its hard error (accepted breaking change). Host suite + go vet green; testops corpus re-verified on the Lima VM (real seccomp + Docker). |
| v0.13.0 | Five RFCs ship together. RFC-040 (Determinism Levels): the new determinism() top-level builtin declares L0/L1 + strict mode; the runtime emits unmediated_io events when the SUT performs I/O Faultbox can observe but isn't mediating (clock_gettime, getrandom, DNS to a non-Faultbox resolver, connect() to an undeclared address). Strict mode (the default) fails the test on the first untolerated leak. RFC-041 (Temporal Properties): eventually(p), always(p, between=), await_event(matcher), await_stable(quiescence_window=), and a rewritten state-machine monitor(name, on=, state_init=, update=, check=) plus a declarative test(name, body=, expect=, timeout=, terminate_when=) builtin. The test lifecycle gains a three-valued verdict (PASS / FAIL / INCONCLUSIVE) with CLI exit code 3 reserved for inconclusive-only runs. RFC-042 (Exploration Plan): faultbox plan subcommand, plan.json in every bundle, the report's Plan tab, coverage analysis (--coverage), rule-based --suggest, --check-cost --max-instances N CI gate, and the body-re-execution engine that turns named choose("name", [opts]) axes (RFC-043 §5.2), syscall-level probability fan-out (max_fires=N, mode="exhaustive"), and parallel(..., interleavings=) orderings into multi-leaf test executions. Each leaf carries a stable LeafID through TestResult, the bundle manifest, and the HTML report's tests table. RFC-043 (Non-deterministic Operators): four small Starlark primitives - choose, nondet, halt(reason) with a new halted outcome, and assume(predicate) / test(assume=) with per-leaf evaluation, AST denylist sandbox, and predicate Starlark errors mapping to Result="error". RFC-044 (Spec Language Simplification): withdraws RFC-013 (param(), superseded by choose()) and RFC-002 (domain(), service()/interface() proved sufficient); unifies the three fan-out axis kinds under one NonDeterministicChoice interface; collapses event sources under observe.stdout/observe.stderr and decoders under decoder("name", ...); deprecates faultbox generate in favor of faultbox plan --suggest. Two new tutorial chapters (Part 4 - Safety & Verification) cover the operator and fan-out vocabulary end-to-end. Full repo go test -race ./... green; Lima sweep 21/21 PASS across the same 6 integration spec suites as v0.12.29. |
| v0.12.29 | Remote services (RFC-036). New service(remote=...) kwarg points Faultbox at an externally-running endpoint - typically a real pod in a customer's k8s dev cluster - without launching a process. The proxy datapath from RFC-024 dials the remote upstream and the SUT reaches it through the proxy unchanged, so every protocol-level fault (response(), error(), slow(), gRPC method targeting, SQL matchers) keeps working. Process-level kwargs (seed=, reset=, reuse=, volumes=, ports=, args=, seccomp=, observe=, ops=) and syscall-level faults are rejected at spec load with explicit error messages naming the offending kwarg and pointing at protocol faults or mock_service(). Composes with RFC-038 tls=tls_cert(...) for TLS-required upstreams (the auto-generated proxy cert covers 127.0.0.1 so SUT-side verification works against the env-rewritten loopback addr). New typed remotes(...) value for services whose interfaces live on different hosts. New @faultbox/discovery/k8s.star stdlib helper exposing k8s.service, k8s.endpoint, and k8s.local - pure string sugar over <name>.<namespace>.svc.cluster.local, no runtime k8s client. The .fb bundle's env.json records every (service, interface, host, protocol) tuple from a remote-using run; faultbox replay warns when replaying such a bundle and points at RFC-037 (the open companion design RFC for the offline-replay determinism story). Cluster connectivity is the user's responsibility - Telepresence connect, kubectl port-forward, in-cluster execution, or VPN - documented in the new Connectivity guide. 49 new tests across spec-load validation (32), runtime/proxy lifecycle (10 incl. TLS×remote interop), bundle round-trip (2), replay warning (2), and string-grep doc gates (3). Full repo go test ./... green; go vet ./... clean; Lima make demo-container 4/4 PASS (proxy datapath refactor confirmed non-regressive against the seccomp + Postgres + Redis path). |