/ docs
On this page

Faultbox is not organized around a mechanism. It is organized around six classes of production bugs - the ones that cause most real outages, live on code paths no ordinary test executes, and are cheap to catch once you can make a dependency fail on demand.

The study behind this framing (“Simple Testing Can Prevent Most Critical Failures”, OSDI 2014) found that the large majority of catastrophic failures in distributed systems came from incorrect handling of ordinary, non-fatal errors - not from exotic race conditions. That is exactly the territory below.

Each class links to the spec pattern that closes it. If you’re new, start with the tutorial - you’ll close the first class within the first session.

Unhandled dependency failure

The incident: the database restarts during a deploy. For ninety seconds every request hits an error path that has never once executed. The API returns raw 500s with stack traces; one handler swallows the error and returns an empty 200; a goroutine leaks per request.

Why nothing caught it: integration tests run against a healthy database - the error branch is dead code until production runs it.

The spec that closes it:

def test_db_down():
    def scenario():
        resp = api.public.post(path="/orders", body='{"item": "widget"}')
        assert_eq(resp.status, 503, "degrade cleanly, never leak internals")
    fault(db, write=deny("EIO"), connect=deny("ECONNREFUSED"), run=scenario)

Every dependency × every operation is a cell in your fault matrix. Most teams discover the matrix is 90% untested cells.

Deep dive: Unhandled dependency failure →

Missing timeout and runaway retry

The incident: a downstream service gets slow - not down, slow. Requests pile up behind a client with no deadline; retries without backoff triple the load; the thread pool drains; a 200ms slowdown in one service becomes a full outage of yours.

Why nothing caught it: slowness is harder to simulate than absence. Unit tests mock the call; integration tests run at full speed.

The spec that closes it:

redis_slow = fault_assumption("redis_slow",
    target = cache.main,
    rules  = [delay(path="/**", delay="800ms")],
)

fault_scenario("slow_cache_bounded",
    scenario = checkout,
    faults   = redis_slow,
    expect   = lambda r: assert_true(r.duration_ms < 2000,
                   "deadline must cap end-to-end latency"),
)

Deep dive: Missing timeout & runaway retry →

Non-idempotent retry

The incident: the payment call times out - after the charge went through. The client retries. The customer is charged twice, and the support ticket has a screenshot.

Why nothing caught it: the bug needs a fault injected between the downstream side effect and the response arriving. No black-box test can place a failure there; Faultbox can, because it sits on the boundary.

The spec that closes it: delay() the response past the caller’s timeout so the operation fires but the caller retries, then assert the side effect happened exactly once - with an invariant, not an eyeball check.

Deep dive: Non-idempotent retry →

Partial failure

The incident: the process crashes between “insert order” and “insert outbox event”. The order exists; the event never fires; the customer paid for something no warehouse will ever ship. Restart hides everything.

Why nothing caught it: the window is milliseconds wide. Only a fault placed deterministically between the two writes opens it every single run.

The spec that closes it: deny the second write (write=deny("EIO", after=1) on the named operation), restart, and assert the invariant - no order row without a matching outbox row - over the real database.

Deep dive: Partial failure →

Failed recovery

The incident: Kafka was down for five minutes at 03:00. It came back. The consumer didn’t - it’s stuck on a poisoned offset, and nobody notices until the Monday backlog graph.

Why nothing caught it: most fault tests check behavior during the fault. Almost none check the system after the fault clears - recovery is the least-tested transition in production systems.

The spec that closes it:

def attempt_during_outage():
    fault(kafka, connect=deny("ECONNREFUSED"), run=lambda:
        api.public.post(path="/orders", body='{"item": "widget"}'))

test("broker_recovery",
    body    = attempt_during_outage,
    expect  = eventually(lambda trace:
        trace.event(type="kafka_publish", topic="orders")),
    timeout = "30s",   # fault cleared - the system must recover within the bound
)

eventually() renders a real verdict: recovered (PASS), didn’t (FAIL), or the test ended before the evidence arrived (INCONCLUSIVE - never a silent green).

Deep dive: Failed recovery →

Bad responses, not outages

The incident: a partner API deploys a change: a field goes from number to string, errors start arriving as 200s with an HTML body. Your parser panics; the panic takes the handler; the handler takes the service.

Why nothing caught it: your mocks are generated from the contract - they’re always well-formed. Reality isn’t.

The spec that closes it:

partner_garbage = fault_assumption("partner_garbage",
    target = partner.main,
    rules  = [response(path="/quote", status=200,
                       body='<html>Bad Gateway</html>')],
)

Response rewriting works on real dependencies and on mock services alike - malformed payloads, wrong status codes, truncated bodies.

Deep dive: Bad responses, not outages →

Where this classification comes from

The six classes are an engineering cut over two decades of production failure studies - not a taxonomy lifted from any single paper. The load-bearing sources:

  • Yuan et al., “Simple Testing Can Prevent Most Critical Failures” (OSDI 2014) - the founding statistic: 92% of catastrophic failures in the studied systems came from incorrect handling of ordinary, non-fatal errors, and in 58% of cases the fault was catchable by simple tests of the error-handling code. Grounds classes 1 and 4.
  • Gunawi et al., “What Bugs Live in the Cloud?” (SoCC 2014) - 3,000+ issues across six production cloud systems; error-handling and recovery bugs dominate. Grounds classes 1 and 5.
  • Gunawi et al., “Why Does the Cloud Stop Computing?” (SoCC 2016) - 597 outages across 32 services: the same pattern seen from the postmortem level.
  • Bailis & Kingsbury, “The Network is Reliable” (CACM 2014) - the evidence that partitions and partial failures are routine operating conditions, not edge cases.
  • Brooker, “Timeouts, retries, and backoff with jitter” (AWS Builders’ Library) - the canonical engineering treatment of class 2.
  • Bronson et al., “Metastable Failures in Distributed Systems” (HotOS 2021) and Huang et al., “Metastable Failures in the Wild” (OSDI 2022) - retry storms as a self-sustaining feedback loop: where class 2 goes when it compounds.
  • Helland, “Idempotence Is Not a Medical Condition” (ACM Queue 2012) - class 3, from the author who named the problem.
  • Pillai et al., “All File Systems Are Not Created Equal” (OSDI 2014) and Zheng et al., “Torturing Databases for Fun and Profit” (OSDI 2014) - crash consistency and torn writes: class 4 below the application.
  • Huang et al., “Gray Failure: The Achilles’ Heel of Cloud-Scale Systems” (HotOS 2017) and Gunawi et al., “Fail-Slow at Scale” (FAST 2018) - degraded-but-not-down components; grounds classes 2 and 6.
  • Alquraan et al., “An Analysis of Network-Partitioning Failures in Cloud Systems” (OSDI 2018) - partition-induced failures are overwhelmingly reproducible with simple event sequences: a fault injector at the boundary reaches them.

Classes 2 and 6 sit at the edge of the gray-failure and metastability literature - that is the direction this catalog grows next.

Using the catalog

Treat these six as a checklist per critical flow: for checkout, which classes have a spec? That question - not “which syscalls can we intercept” - is the review conversation Faultbox is built for. The methodology guide turns it into a process; the fault matrix enumerates it; reports make the answer visible in the PR.

Edit this page on GitHub