/ docs
On this page

Bug Class 2: Missing Timeout & Runaway Retry

Slow is more dangerous than down. Absence produces an error you might handle; slowness produces nothing at all - requests queue, pools drain, and a 200ms hiccup in one service becomes a full outage of yours.

The incident

A downstream API’s p99 goes from 40ms to 900ms - not down, just slow. Your client has no deadline, so every in-flight request waits. Your retry policy has no backoff, so load triples. The thread pool saturates; now your callers see the same slowness and retry too. The postmortem calls it a “retry storm”; the root cause is two missing lines: a timeout and a backoff.

Why your tests miss it

Slowness is harder to simulate than absence - unit tests mock the call and return instantly, integration tests run at full speed. Nobody’s test suite includes a dependency that answers in 900ms.

The spec that closes it

partner_slow = fault_assumption("partner_slow",
    target = partner.main,
    rules  = [delay(path = "/quote*", delay = "2s")],
)

fault_scenario("checkout_bounded_latency",
    scenario = checkout,
    faults   = partner_slow,
    expect   = lambda r: (
        assert_eq(r.status, 200, "degrade or fail fast - never hang"),
        assert_true(r.duration_ms < 500,
            "your latency must not track the dependency's latency"),
    ),
)

And the retry-amplification check - the fault fires, but the number of attempts stays bounded:

def bounded_retries(trace):
    return trace.count(match.event(type = "http_request", path = "/quote*")) <= 3

always(bounded_retries)

What to assert

  • End-to-end latency bounds, not just status codes - assert_true on r.duration_ms is the assertion most suites never make.
  • Attempt counts under the fault: retries with backoff and a cap, not amplification.
  • What the caller gets when the deadline fires: a clean timeout error, not a hang.

Which level

Protocol - the slowness must hit one operation while the service and its healthcheck stay healthy. Syscall connect=delay() is the blunter variant for “the whole network is slow”. Details in Choosing Fault Levels.

Learn the pieces

Edit this page on GitHub