Features
v0.18.0 · Apache 2.0 · Linux (macOS via Lima)
Protocol-Level Fault Injection
Inject faults at the protocol level via transparent proxy. Target specific HTTP paths, SQL queries, Redis commands, or Kafka topics - without touching the network stack.
fault(service) = syscall level. fault(service.interface) = protocol
level. Same builtin, different dispatch.
Return HTTP 503 for POST /orders, inject Postgres query errors,
drop Kafka messages on specific topics.
Mock Services - Simulate What You Can't Run
The dependencies you don't own - another team's API, a paid SaaS, an auth provider - become in-process, protocol-aware stand-ins. No containers, no images, first test in under 100 ms. And they accept the same fault rules as real services.
mock_service() for HTTP, HTTP/2, gRPC, TCP, UDP;
redis.server(state=...), kafka.broker(topics=...),
mongo.server(collections=...) for infrastructure.
http.server(openapi="petstore.yaml") - response stubs
generated from the OpenAPI document; typed gRPC mocks from your protos.
The proxy layer rewrites mock responses with the same
error(), response(), delay() rules -
one fault vocabulary across mocks and real dependencies.
Reports - Where the Analysis Happens
Every run writes a self-contained .fb bundle: spec,
environment, full event trace, replay script. One command -
faultbox report run.fb - turns it into a single HTML
file that opens anywhere: swim-lane trace per service, per-test
verdicts with drill-down, the plan tree and its coverage. The
terminal says PASS or FAIL; the report is where you understand
why.
CSS, JS, and compressed trace data inlined - no server, no network. Attach it to a PR, an incident ticket, or an email.
Every service a lane, every intercepted operation an event - fault decisions and causality visible at a glance.
faultbox replay run.fb re-executes the exact
fault schedule that produced the report.
Deterministic Exploration
Name the unknowns in your spec with choose() and nondet(),
and Faultbox expands them into a tree of test executions - one leaf per
fault, ordering, and choice. faultbox plan shows the tree before
you run it; every failing leaf replays from its seed.
choose("backend", [primary, replica]) and
parallel(..., interleavings=...) fan out into multiple leaves,
each with a stable LeafID in the report.
faultbox plan --coverage analyses the tree statically - no
services launched. --check-cost --max-instances N guards blow-ups.
Every leaf has a seed. Failed? Replay with --seed 42 for an
identical interleaving - deterministic debugging.
Starlark Specs
Topology, faults, and assertions in one .star file.
Starlark is a Python dialect - if you know Python, you know Starlark.
No YAML, no separate config language. The spec is executable code.
service(), interface(), depends_on,
healthcheck - declare topology as code.
assert_eq(), assert_eventually(),
assert_never(), assert_before() - value checks
and temporal properties on the syscall trace.
Register happy paths with scenario(), then
faultbox generate creates failure tests automatically.
Event Log & Traces
Every intercepted syscall is recorded with vector clocks, service attribution, and file paths. Assert on internal behavior, not just inputs and outputs.
"The WAL write happened before the response" -
assert_before() proves ordering guarantees.
--shiviz trace.shiviz produces a space-time diagram
with causal arrows between services.
Capture before/after a refactor. faultbox diff shows
exactly what behavioral changes you introduced.
Binary & Container Modes
Run local binaries for fast development, or real infrastructure in Docker containers for integration testing. Same spec, same assertions, same faults.
binary="./my-service" - fork+exec with seccomp filter.
Fastest iteration, no Docker needed.
image="postgres:16" - Docker containers with faultbox-shim
entrypoint. Test against real Postgres, Redis, Kafka.
Packet-Level Network Faults
The layer between a syscall and a parsed protocol message. A
dropped packet sends no RST - so the socket stays in
ESTABLISHED, writing into a void until a keepalive fires.
That is the failure that actually takes production down, and
drop() could never reach it, because closing a connection
gives well-written clients an ECONNRESET they handle on the
first try.
packet_drop, packet_delay,
packet_reorder, packet_duplicate,
packet_corrupt, packet_reset,
packet_window, packet_pass - matched on
direction, TCP flags, port, length, or payload prefix.
bandwidth("1mbit", queue="250ms") and
mtu(576) describe the link rather than the packet. The
queue is bounded in time, so the shaper drops when saturated
the way a real bottleneck does.
Opt in with determinism(runtime = "gvisor"). Linux
with CAP_NET_ADMIN; use the Lima VM on macOS.
Contract-Driven Clients
Faultbox reads your OpenAPI and protobuf contracts to build the
dependencies you can't run. client() reads them from
the other side, to build the callers that drive your service - and makes
each caller a named actor in the trace instead of one anonymous
test lane.
client("mobile-app", target=orders.public, openapi="./orders.yaml")
then mobile.get_order(order_id=42) - no path, no verb, no
field names.
validate="response" checks each response against the
schema declared for its status code. It records the verdict and emits
a contract_violation event rather than raising - under
fault, the violation is usually the finding.
client_call / client_return land on the
client's own swim lane with its own vector-clock participant, and
work as temporal anchors with no new matcher syntax.
Filesystem Observation
watch() reports a service's file I/O with
resolved paths. Faultbox could always count
write calls; it could not reliably say which file,
because a syscall carries a descriptor and resolving it meant reading
/proc out of band, racing the SUT.
"never wrote outside its data directory" - the canonical assertion is negative, and only as strong as the trace behind it.
A run fails when no sandbox connected, when trace points were dropped during the window, when points matched no launched service, or on a decode error - rather than passing on an empty observation.
A trace point fires after the syscall, and gVisor has no
fsync point - so write ordering is provable and
durability is not. ops=["fsync"] is rejected rather than
quietly returning nothing.
Monitors & Network Partitions
Define safety invariants as monitors that run on every syscall event. Simulate network partitions between specific services.
Callbacks that fire on every matching event - fail immediately if an invariant is violated.
partition(orders, inventory, run=scenario) - built on
the packet gateway, with direction= for one-way splits
and partition_start() / partition_stop() for
timing. It cuts established connections, not just
connect(), so it holds against a service that pools
them.
Named Operations
Group related syscalls into logical operations. Fault "persist" instead of "write + fsync". Path filters target specific files.
ops={"persist": op(syscalls=["write","fsync"], path="*.wal")} -
fault the WAL persist operation, not individual syscalls.
Recipe Library
Curated, protocol-specific failure wrappers ship embedded in the
faultbox binary. Load them via the @faultbox/
prefix - no filesystem setup, no network fetch. Each recipe encodes a
canonical error message, status code, or incident pattern drawn from
real postmortems.
load("@faultbox/recipes/mongodb.star", "mongodb") -
works from any project, no recipes/ directory needed.
Ships with every binary.
mongodb.disk_full() and postgres.disk_full()
coexist. One import per protocol, zero name collisions.
Say cassandra.unavailable() instead of remembering
"Cannot achieve consistency level QUORUM". Recipes stay in sync
with real driver behavior.
faultbox recipes list and recipes show <name>
- browse the catalog without reading source.
Under the Hood: Syscall-Level Fault Injection
The engine beneath it all: intercept any syscall via Linux's seccomp-notify and decide - allow, deny, or delay. No eBPF, no ptrace, no code changes. Works on any binary: Go, Rust, Java, Python, C.
write automatically covers write, writev,
pwrite64. Think in operations, not syscall numbers.
Fault only writes to /data/*.wal - stdout, TCP, and other writes
are unaffected. Powered by fd→path resolution via /proc.
deny("EIO", probability="30%") or deny("EIO", trigger="after=5") -
intermittent failures and trigger-on-Nth-call.
LLM-First Design
New in v0.2.0
Faultbox is designed for both human engineers and LLM agents. Structured JSON output, MCP server, Claude Code integration - everything an agent needs for an autonomous code → test → fix loop.
faultbox mcp - 6 tools for Claude, Cursor, and any MCP client.
Run tests, generate specs, analyze failures natively.
--format json - machine-parseable results with fault info,
syscall summary, and actionable diagnostics.
faultbox init --claude - slash commands (/fault-test,
/fault-generate, /fault-diagnose) and auto-MCP config.
faultbox init --from-compose - zero-effort spec generation.
Detects protocols, wires dependencies, generates happy-path tests.
Not just "test failed" - structured hints like "write fault fired but service returned 200 - missing error handling in persist path."
ghcr.io/faultbox/faultbox image + GitHub Action for
automated fault testing on every PR.
Recent Releases
| Version | Highlights |
|---|---|
| v0.18.0 current | Field-report fixes from the first onboarding of a large production Go service. Seven reported issues, plus one found while building the regression corpus for the first of them. Two of the reports - a host-binary SUT losing every outbound socket, and the proxy data plane freezing at the test-phase boundary - were one defect, and it was not the proxy. SECCOMP_IOCTL_NOTIF_RECV returns ENOENT when a notification has already been discarded, which says nothing about the listener fd; the classifier matched it anyway by substring-searching the errno text, so the notification loop returned nil on the first one and stopped supervising for good - silently, because a nil error reads as a clean shutdown. The child kept its seccomp filter with nobody to answer it, so every intercepted syscall blocked forever. That accounts for reads and new dials both dying, for it never recovering, and for a busy connection pool always losing while a quiet client survived. A loop that ends while its target is alive now fails the test loudly. A fault written the normal way now installs a filter - multi-line calls and spaces around = matched neither literal substring the scan looked for - and where the scan still cannot see a fault, the runtime refuses it (FAULT_NOT_FILTERABLE) rather than installing something inert, so a fault either fires or the run fails saying why. mock_service() works from a containerized SUT; the Docker network survives the whole run (it was destroyed after each test and never recreated, so from test 2 every container landed on the default bridge with no name resolution); timeouts are no longer blamed on faults that were not there; replay loads specs in a subdirectory and prints a hint it accepts; packet faults reach a containerized consumer. Also grpc.call() completed its first round trip against a real server, and all 13 step protocols now have a real-server spec. |
| v0.17.0 | faultbox check, and diagnostics for suites that cannot fail. faultbox check spec.star validates a spec without launching a process, pulling an image, or needing Docker - milliseconds against the tens of seconds a run costs. The runtime could always do this; it simply was not exposed, so the only way to learn a spec was malformed was to run it. Findings carry machine-readable codes and a suggested next move, and the same code backs the MCP tool check_spec - a check that behaved differently through MCP would give an agent a wrong model of the tool. The two new diagnostics come from an uncomfortable finding: a CI spec exercised a broken Postgres client on every pull request for three releases and passed, because its only assertion was that a query fails under an injected fault - which a client that cannot connect at all satisfies identically. Its own comment stated the intent, and the care is what hid the bug. NO_POSITIVE_CONTROL fires when an interface is stepped but no test ever asserts a step on it succeeds; it is suite-level, which is what makes it new, since no per-test lint can see it. TEST_NO_ASSERTIONS fires when a test passed having evaluated nothing. |
| v0.16.1 | The protocol audit: eleven fixes, ten of them pre-existing. v0.16.0 fixed two Postgres bugs that survived because no spec had ever asserted on the result of a database step; this release repeats that audit across the other twelve protocols - one spec per protocol, a real server, every result checked. ready() could never succeed for eight protocols (a v0.16.0 regression - the check resolved to redis://host:port and eight plugins dialled that string verbatim, burning the whole timeout; Redis went from a 60 s failure to a 407 ms pass). MySQL steps could not authenticate or select a database, and once the credential fix let a step reach the command phase for the first time it exposed a second bug: the MySQL proxy hung forever on every result set, and because Stop() waited unbounded on the connection WaitGroup, one stuck handler hung the entire run after the test body had finished. All twelve proxies now bound teardown at 5 s and emit proxy_stop_timeout rather than hanging silently. Also fixed: args= was accepted by the spec loader and silently dropped for container services; dict and list step arguments were mangled (every int, float, bool and nested value inside a dict became "", self-consistently enough that round trips looked correct); the NATS proxy corrupted every line it forwarded by mixing bufio.Scanner line-stripping with \n-only writes against a CRLF protocol; NATS publish reported success without confirming delivery; ready() made a single protocol attempt for MongoDB and Cassandra at the moment the server is least likely to be up; and every container leaked its anonymous volume - measured at 290 orphaned volumes and 18.7 GB on the dev VM, which filled a 30 GB disk and surfaced as a flaky test three specs later. New: credentials read from env= for MySQL, Redis, MongoDB, ClickHouse, Cassandra and NATS, resolved in one place so healthchecks and steps authenticate identically. Ten of thirteen step protocols now have real-server coverage; http2, udp and grpc remain unit-tested only. Six consecutive clean go test -race ./... full-suite runs. |
| v0.16.0 | Filesystem observation (RFC-056). New watch(service, files=, ops=, run=) reports a service's file I/O with resolved paths. Faultbox could already count write calls but could not reliably say which file - a syscall carries a descriptor, and resolving it meant reading /proc out of band, racing the SUT. This closes fs-unmediated, a determinism category that had emitted no events since RFC-040, so file I/O outside declared paths was silently undetected. Built in v0.14.0 and withdrawn before release: runsc trace create instruments only tasks created after it attaches, so a network-driven query against a running Postgres produced 2 trace points where the same SQL from a freshly spawned process produced 1054 - a watch that observes nothing still runs, and every assertion under it still passes. --pod-init-config installs the session at sandbox boot instead: 236 points on that same workload. Completeness is enforced, not assumed - the canonical assertion is negative ("never wrote outside its data directory") and only as strong as the trace behind it, so a run fails when no sandbox connected, when points were dropped during the window, when points matched no launched service, or on a decode error. New faultbox setup-trace does the one-time host registration (the flag lives in daemon.json, not per container) - idempotent, reports what it left alone, and prints the Docker restart rather than performing it. New ready(timeout=), protocol-aware readiness: tcp() asks whether something is listening, which for a container is true the moment Docker's port proxy binds - measured at 0 ms against a Postgres needing ~10 s. ready() asks the service instead, via its interface's protocol plugin and declared credentials; on the RFC-056 corpus it replaced a hand-tuned 25-second sleep() with a ~2.4 s check that is correct by construction. Fixed: Postgres steps could never authenticate - the connection string carried no credentials (lib/pq fell back to the OS user, root under sudo), and the auth handshake was relayed server→client only, so SCRAM-SHA-256 (the postgres:14+ default), MD5 and cleartext all deadlocked until the client's 60-second read deadline. Observation only: gVisor has no fsync point, so write ordering is provable and durability is not - ops=["fsync"] is rejected rather than quietly returning nothing. |
| v0.15.0 | Contract-driven clients (RFC-055). Faultbox already read your OpenAPI and protobuf contracts to build the dependencies you can't run; this release reads them from the other side, to build the callers that drive your service - and makes each caller a named actor in the trace rather than one anonymous test lane. New client() turns an OpenAPI 3.x document or a protobuf FileDescriptorSet into a named caller bound to a service interface, with its operations generated as callable attributes: mobile = client("mobile-app", target = orders.public, openapi = "./orders.yaml", validate = "response"), then mobile.get_order(order_id = 42) - no path, no verb, no field names. The half that isn't ergonomics: with the contract loaded caller-side, "under fault, the service still returned what it published" becomes a one-kwarg check. validate="response" checks each response against the schema declared for its status code, records the verdict on resp.contract_ok / resp.contract_error, and emits a contract_violation event - it deliberately does not raise, because a contract violation under fault is usually the finding, not a harness error, and an undeclared status code counts as a violation, which is how the undocumented degraded path surfaces. Clients are first-class trace actors: calls emit client_call / client_return on the client's own swim lane with its own vector-clock participant, and those events work as temporal anchors with no new matcher syntax. Client calls dial through the same proxy resolution as step methods, so fault(iface, ...) applies unchanged; TLS (RFC-038) and remote targets (RFC-036) are inherited from the interface. Clients are trace actors, not processes - no seccomp filter, never a fault target. New faultbox inspect --clients prints each client's generated operation table. interface(..., spec=) is now read - parsed and stored since early versions with nothing consuming it, it is now inherited by a client() that declares no contract of its own. Fixed: mock_service(openapi=)/(descriptors=) resolved relative paths against the process working directory rather than the spec's own directory; service() and client() now reject a name already taken by the other, since sharing one folded two participants into a single lane and vector clock. v1 is unary-only and stateless per call. |