Features
v0.19.0 · Apache 2.0 · Linux (macOS via Lima)
Protocol-Level Fault Injection
Inject faults at the protocol level via transparent proxy. Target specific HTTP paths, SQL queries, Redis commands, or Kafka topics - without touching the network stack.
fault(service) = syscall level. fault(service.interface) = protocol
level. Same builtin, different dispatch.
Return HTTP 503 for POST /orders, inject Postgres query errors,
drop Kafka messages on specific topics.
Mock Services - Simulate What You Can't Run
The dependencies you don't own - another team's API, a paid SaaS, an auth provider - become in-process, protocol-aware stand-ins. No containers, no images, first test in under 100 ms. And they accept the same fault rules as real services.
mock_service() for HTTP, HTTP/2, gRPC, TCP, UDP;
redis.server(state=...), kafka.broker(topics=...),
mongo.server(collections=...) for infrastructure.
http.server(openapi="petstore.yaml") - response stubs
generated from the OpenAPI document; typed gRPC mocks from your protos.
The proxy layer rewrites mock responses with the same
error(), response(), delay() rules -
one fault vocabulary across mocks and real dependencies.
Reports - Where the Analysis Happens
Every run writes a self-contained .fb bundle: spec,
environment, full event trace, replay script. One command -
faultbox report run.fb - turns it into a single HTML
file that opens anywhere: swim-lane trace per service, per-test
verdicts with drill-down, the plan tree and its coverage. The
terminal says PASS or FAIL; the report is where you understand
why.
CSS, JS, and compressed trace data inlined - no server, no network. Attach it to a PR, an incident ticket, or an email.
Every service a lane, every intercepted operation an event - fault decisions and causality visible at a glance.
faultbox replay run.fb re-executes the exact
fault schedule that produced the report.
Deterministic Exploration
Name the unknowns in your spec with choose() and nondet(),
and Faultbox expands them into a tree of test executions - one leaf per
fault, ordering, and choice. faultbox plan shows the tree before
you run it; every failing leaf replays from its seed.
choose("backend", [primary, replica]) and
parallel(..., interleavings=...) fan out into multiple leaves,
each with a stable LeafID in the report.
faultbox plan --coverage analyses the tree statically - no
services launched. --check-cost --max-instances N guards blow-ups.
Every leaf has a seed. Failed? Replay with --seed 42 for an
identical interleaving - deterministic debugging.
Starlark Specs
Topology, faults, and assertions in one .star file.
Starlark is a Python dialect - if you know Python, you know Starlark.
No YAML, no separate config language. The spec is executable code.
service(), interface(), depends_on,
healthcheck - declare topology as code.
assert_eq(), assert_eventually(),
assert_never(), assert_before() - value checks
and temporal properties on the syscall trace.
Register happy paths with scenario(), then
faultbox generate creates failure tests automatically.
Event Log & Traces
Every intercepted syscall is recorded with vector clocks, service attribution, and file paths. Assert on internal behavior, not just inputs and outputs.
"The WAL write happened before the response" -
assert_before() proves ordering guarantees.
--shiviz trace.shiviz produces a space-time diagram
with causal arrows between services.
Capture before/after a refactor. faultbox diff shows
exactly what behavioral changes you introduced.
Binary & Container Modes
Run local binaries for fast development, or real infrastructure in Docker containers for integration testing. Same spec, same assertions, same faults.
binary="./my-service" - fork+exec with seccomp filter.
Fastest iteration, no Docker needed.
image="postgres:16" - Docker containers with faultbox-shim
entrypoint. Test against real Postgres, Redis, Kafka.
Packet-Level Network Faults
The layer between a syscall and a parsed protocol message. A
dropped packet sends no RST - so the socket stays in
ESTABLISHED, writing into a void until a keepalive fires.
That is the failure that actually takes production down, and
drop() could never reach it, because closing a connection
gives well-written clients an ECONNRESET they handle on the
first try.
packet_drop, packet_delay,
packet_reorder, packet_duplicate,
packet_corrupt, packet_reset,
packet_window, packet_pass - matched on
direction, TCP flags, port, length, or payload prefix.
bandwidth("1mbit", queue="250ms") and
mtu(576) describe the link rather than the packet. The
queue is bounded in time, so the shaper drops when saturated
the way a real bottleneck does.
Opt in with determinism(runtime = "gvisor"). Linux
with CAP_NET_ADMIN; use the Lima VM on macOS.
Contract-Driven Clients
Faultbox reads your OpenAPI and protobuf contracts to build the
dependencies you can't run. client() reads them from
the other side, to build the callers that drive your service - and makes
each caller a named actor in the trace instead of one anonymous
test lane.
client("mobile-app", target=orders.public, openapi="./orders.yaml")
then mobile.get_order(order_id=42) - no path, no verb, no
field names.
validate="response" checks each response against the
schema declared for its status code. It records the verdict and emits
a contract_violation event rather than raising - under
fault, the violation is usually the finding.
client_call / client_return land on the
client's own swim lane with its own vector-clock participant, and
work as temporal anchors with no new matcher syntax.
Filesystem Observation
watch() reports a service's file I/O with
resolved paths. Faultbox could always count
write calls; it could not reliably say which file,
because a syscall carries a descriptor and resolving it meant reading
/proc out of band, racing the SUT.
"never wrote outside its data directory" - the canonical assertion is negative, and only as strong as the trace behind it.
A run fails when no sandbox connected, when trace points were dropped during the window, when points matched no launched service, or on a decode error - rather than passing on an empty observation.
A trace point fires after the syscall, and gVisor has no
fsync point - so write ordering is provable and
durability is not. ops=["fsync"] is rejected rather than
quietly returning nothing.
Monitors & Network Partitions
Define safety invariants as monitors that run on every syscall event. Simulate network partitions between specific services.
Callbacks that fire on every matching event - fail immediately if an invariant is violated.
partition(orders, inventory, run=scenario) - built on
the packet gateway, with direction= for one-way splits
and partition_start() / partition_stop() for
timing. It cuts established connections, not just
connect(), so it holds against a service that pools
them.
Named Operations
Group related syscalls into logical operations. Fault "persist" instead of "write + fsync". Path filters target specific files.
ops={"persist": op(syscalls=["write","fsync"], path="*.wal")} -
fault the WAL persist operation, not individual syscalls.
Recipe Library
Curated, protocol-specific failure wrappers ship embedded in the
faultbox binary. Load them via the @faultbox/
prefix - no filesystem setup, no network fetch. Each recipe encodes a
canonical error message, status code, or incident pattern drawn from
real postmortems.
load("@faultbox/recipes/mongodb.star", "mongodb") -
works from any project, no recipes/ directory needed.
Ships with every binary.
mongodb.disk_full() and postgres.disk_full()
coexist. One import per protocol, zero name collisions.
Say cassandra.unavailable() instead of remembering
"Cannot achieve consistency level QUORUM". Recipes stay in sync
with real driver behavior.
faultbox recipes list and recipes show <name>
- browse the catalog without reading source.
Under the Hood: Syscall-Level Fault Injection
The engine beneath it all: intercept any syscall via Linux's seccomp-notify and decide - allow, deny, or delay. No eBPF, no ptrace, no code changes. Works on any binary: Go, Rust, Java, Python, C.
write automatically covers write, writev,
pwrite64. Think in operations, not syscall numbers.
Fault only writes to /data/*.wal - stdout, TCP, and other writes
are unaffected. Powered by fd→path resolution via /proc.
deny("EIO", probability="30%") or deny("EIO", trigger="after=5") -
intermittent failures and trigger-on-Nth-call.
LLM-First Design
New in v0.2.0
Faultbox is designed for both human engineers and LLM agents. Structured JSON output, MCP server, Claude Code integration - everything an agent needs for an autonomous code → test → fix loop.
faultbox mcp - 6 tools for Claude, Cursor, and any MCP client.
Run tests, generate specs, analyze failures natively.
--format json - machine-parseable results with fault info,
syscall summary, and actionable diagnostics.
faultbox init --claude - slash commands (/fault-test,
/fault-generate, /fault-diagnose) and auto-MCP config.
faultbox init --from-compose - zero-effort spec generation.
Detects protocols, wires dependencies, generates happy-path tests.
Not just "test failed" - structured hints like "write fault fired but service returned 200 - missing error handling in persist path."
ghcr.io/faultbox/faultbox image + GitHub Action for
automated fault testing on every PR.
Recent Releases
| Version | Highlights |
|---|---|
| v0.19.0 current | Environment diagnostics and Kafka consumer barriers. Read-only faultbox doctor checks runtime prerequisites, Docker/shim pairing and host/Lima versions with actionable text or JSON results. A minimal macOS runner profile includes Docker without a mandatory language toolchain. Native Kafka mock wait_ready and wait_committed steps enforce current assignments, process identity and test-scoped publish receipts. Missing topics produce concise diagnostics; stale and ambiguous evidence cannot pass. A commit remains distinct from business completion. |
| v0.18.2 | Repeatable service harnesses. Kafka readiness and commit evidence identify the actual managed Linux process, so two consumers can share a client ID without borrowing each other's position or waiting on application logs. Native grpc.call(descriptors=..., body=...) sends typed protobuf requests without reflection. Per-test mock profiles are applied before services boot; consume_many provides bounded typed Kafka observation. Portable resources and module-relative paths survive replay. Startup detects occupied ports and early exits; teardown is bounded and reports failures. The shared-client-ID regression passed 12/12 checks without warm-up records or log fallback. |
| v0.18.1 | Kafka determinism, and the prerequisites that were never written down. Kafka-consuming specs were non-deterministic at a fixed seed, and the seed could never have fixed them - the state was in the broker, not in Faultbox. Both consumer paths used the constant group ID "faultbox", and a group's committed offsets live in __consumer_offsets and outlive the reader that wrote them, so the second test to consume resumed after whatever the first had committed; a broker container kept across tests - normal, since brokers are slow to start - carried that state between whole runs. The default group is now scoped to (run, test), which has committed nothing, so the read starts at the beginning every time, while a spec that is about consumer-group semantics passes group= and still gets a stable name. It landed in two commits and the second is the one worth recording: the first claimed to have scoped both consumer paths and had scoped one, leaving the topic event source on the constant. Measured against apache/kafka:3.7.0 with a reused broker: two tests that each publish then consume both read offset 0. This makes runs reproducible; it does not isolate topic contents between tests, which needs a per-test topic. Documentation caught up with three things it had left implicit: packet faults need root (creating a TUN device is privileged, so sudo or a one-time setcap), the two gVisor-backed features do not need the same prerequisites - packet faults want CAP_NET_ADMIN and /dev/net/tun but not runsc, watch() wants runsc and a host registration but no elevated capability - and a Kafka fault rule may never match because the client asks the broker where it is and reconnects to that address, which is fixed by pointing KAFKA_ADVERTISED_LISTENERS at the late-bound kafka.main.proxy_addr. RFC-057 scopes the general fix for protocols that advertise their own address. |
| v0.18.0 | Field-report fixes from the first onboarding of a large production Go service. Seven reported issues, plus one found while building the regression corpus for the first of them. Two of the reports - a host-binary SUT losing every outbound socket, and the proxy data plane freezing at the test-phase boundary - were one defect, and it was not the proxy. SECCOMP_IOCTL_NOTIF_RECV returns ENOENT when a notification has already been discarded, which says nothing about the listener fd; the classifier matched it anyway by substring-searching the errno text, so the notification loop returned nil on the first one and stopped supervising for good - silently, because a nil error reads as a clean shutdown. The child kept its seccomp filter with nobody to answer it, so every intercepted syscall blocked forever. That accounts for reads and new dials both dying, for it never recovering, and for a busy connection pool always losing while a quiet client survived. A loop that ends while its target is alive now fails the test loudly. A fault written the normal way now installs a filter - multi-line calls and spaces around = matched neither literal substring the scan looked for - and where the scan still cannot see a fault, the runtime refuses it (FAULT_NOT_FILTERABLE) rather than installing something inert, so a fault either fires or the run fails saying why. mock_service() works from a containerized SUT; the Docker network survives the whole run (it was destroyed after each test and never recreated, so from test 2 every container landed on the default bridge with no name resolution); timeouts are no longer blamed on faults that were not there; replay loads specs in a subdirectory and prints a hint it accepts; packet faults reach a containerized consumer. Also grpc.call() completed its first round trip against a real server, and all 13 step protocols now have a real-server spec. |
| v0.17.0 | faultbox check, and diagnostics for suites that cannot fail. faultbox check spec.star validates a spec without launching a process, pulling an image, or needing Docker - milliseconds against the tens of seconds a run costs. The runtime could always do this; it simply was not exposed, so the only way to learn a spec was malformed was to run it. Findings carry machine-readable codes and a suggested next move, and the same code backs the MCP tool check_spec - a check that behaved differently through MCP would give an agent a wrong model of the tool. The two new diagnostics come from an uncomfortable finding: a CI spec exercised a broken Postgres client on every pull request for three releases and passed, because its only assertion was that a query fails under an injected fault - which a client that cannot connect at all satisfies identically. Its own comment stated the intent, and the care is what hid the bug. NO_POSITIVE_CONTROL fires when an interface is stepped but no test ever asserts a step on it succeeds; it is suite-level, which is what makes it new, since no per-test lint can see it. TEST_NO_ASSERTIONS fires when a test passed having evaluated nothing. |