Bug Class 3: Non-Idempotent Retry
The retry that succeeds twice. The failure lands after the side effect happened but before the response arrived - so the caller retries an operation that already worked.
The incident
The payment call times out. The charge went through - the timeout hit on the way back. The client retries; the customer is charged twice; the support ticket arrives with a screenshot and the refund costs more than the order.
The same shape hides everywhere: duplicate emails, double-shipped orders, twice-applied coupons, replayed webhooks.
Why your tests miss it
The bug lives in a milliseconds-wide window between the downstream side effect and the response. No black-box test can place a failure there - you can’t make a real payment gateway succeed-then-timeout on demand. Faultbox can, because it sits on the boundary and controls exactly when the response comes back.
The spec that closes it
Delay the charge response past the caller’s timeout - the charge fires server-side, the caller gives up and retries; then assert the side-effect count:
def test_charge_idempotent():
# First attempt: delay the charge response past the caller's timeout,
# so the charge itself fires but the caller sees a 504 and retries.
def first_attempt():
resp = api.public.post(path = "/checkout", body = '{"cart": 42}')
assert_eq(resp.status, 504) # caller saw a timeout
fault(payments.main, delay(delay = "10s"), run = first_attempt)
# Fault cleared - the retry goes through cleanly:
resp = api.public.post(path = "/checkout", body = '{"cart": 42}')
assert_eq(resp.status, 201) # retry succeeds
charges = events(service = "payments", op = "charge")
assert_eq(len(charges), 1, "one cart, one charge - ever")What to assert
- The side-effect count, from the trace - not the response body. The response says what your service believes; the trace says what actually happened at the boundary.
- Idempotency keys actually flow: same key on the retry, and the downstream sees it.
Which level
Protocol delay() past the caller’s timeout plus the retry at the API
level - the failure must be placed between the side effect and the
response, which is what delaying the reply gives you.
Named operations make the
target readable: delay the "charge" operation, not write on fd 7.