/ docs
On this page

Bug Class 1: Unhandled Dependency Failure

The most common class behind production incidents: a dependency returns an ordinary error - connection refused, EIO, a 503 - and your code’s error branch, which has never once executed, does the wrong thing.

The incident

The database restarts during a deploy. For ninety seconds every request hits an error path that no test has ever run. One handler returns a raw 500 with a stack trace. Another swallows the error and returns an empty 200 - the client caches it. A third leaks a goroutine per request; the service OOMs eight minutes after the database is already back.

Why your tests miss it

Integration tests run against a healthy database. Unit tests mock the call and return the error you thought of - not the one the driver actually produces. The error branch is dead code until production executes it for you.

The spec that closes it

Every dependency, every operation, both failure shapes - refused and erroring:

db_down = fault_assumption("db_down",
    target  = db,
    connect = deny("ECONNREFUSED"),
)

insert_fails = fault_assumption("insert_fails",
    target = db.main,
    rules  = [error(query = "INSERT INTO orders*")],
)

fault_matrix(
    scenarios = [place_order, get_order, health],
    faults    = [db_down, insert_fails],
    overrides = {
        (place_order, db_down): lambda r: assert_eq(r.status, 503,
            "degrade cleanly - no stack traces, no fake 200s"),
        (health, db_down): lambda r: assert_eq(r.status, 200,
            "healthcheck must not depend on the database"),
    },
)

What to assert

  • Clean, correct status codes - a 5xx that says “unavailable”, never a 200 that lies and never internals leaking to the caller.
  • The healthcheck stays green when a dependency is down - otherwise one dead database restarts your whole fleet.
  • No resource leaks: trace.count() connections opened vs closed under the fault.

Which level

Both, in sequence: syscall connect=deny() for “the whole dependency is gone”, protocol error() for “this one operation is refused” - see Choosing Fault Levels. Most teams discover their fault matrix is 90% untested cells; start with the one dependency your critical flow can least afford to lose.

Learn the pieces

Edit this page on GitHub