/ docs
On this page

Bug Class 5: Failed Recovery

The dependency came back; the system didn’t. Recovery is the least-tested transition in production systems - almost every fault test checks behavior during the fault, almost none checks the system after the fault clears.

The incident

Kafka was down for five minutes at 03:00. It came back on its own; the on-call never woke up. But the consumer is stuck - dead connection pool, poisoned offset, a circuit breaker that never half-opens. Nobody notices until Monday, when the backlog graph has been climbing for sixty hours and the “real-time” dashboard is three days stale.

Why your tests miss it

Recovery bugs live in the transition, not in either steady state. Testing “works normally” passes; testing “fails when broker is down” passes; the stuck-forever path between them is exercised by neither. And in production the transition happens at 03:00, unobserved.

The spec that closes it

Scope the fault, let it clear, then make recovery itself the assertion - with a temporal property, not a sleep:

def attempt_during_outage():
    # during the fault: deferred, not lost. The fault clears the moment
    # run= returns, so recovery is what the eventually() below watches for.
    fault(kafka, connect = deny("ECONNREFUSED"), run = lambda:
        api.public.post(path = "/orders", body = '{"cart": 42}'))

test("broker_recovery",
    body    = attempt_during_outage,
    expect  = eventually(lambda trace:
        trace.event(type = "kafka_publish", topic = "orders")),
    timeout = "30s",   # the bound: recovery must arrive within it
)

The verdict semantics matter here: eventually() renders PASS only if the evidence arrives, FAIL if the run completes without it, and INCONCLUSIVE if the test ended before the question was decided - a timeout is never a silent green.

What to assert

  • The deferred work actually flows after the fault clears - events published, queue drained, cache repopulated.
  • Recovery happens within a bound (the test’s timeout=), because “eventually, eventually” is an outage with better marketing.
  • Steady-state resumes: post-recovery requests behave identically to pre-fault ones (compare traces).

Which level

Either level for the fault - what defines this class is the scoped fault (it must end) plus the temporal assertion after it. See Choosing Fault Levels.

Learn the pieces

Edit this page on GitHub