/ docs
On this page

Bug Class 4: Partial Failure & Torn Writes

The crash that lands between two writes that were supposed to be one. Half the state exists; the other half never will; restart hides the evidence.

The incident

The process dies between “insert order” and “insert outbox event”. The order row exists; the event never fires; the customer paid for something no warehouse will ever ship. Nobody notices - the service restarted cleanly, the dashboards are green, and the orphan row sits in the table until a reconciliation job (or a customer email) finds it weeks later.

Why your tests miss it

The window is milliseconds wide and depends on where exactly the failure lands. Random crash testing hits it once per thousand runs; a fault placed deterministically between the two writes opens it every single run.

The spec that closes it

Fail the outbox write so the transaction can’t complete cleanly - deny("EIO") on the named operation:

torn_write = fault_assumption("torn_write",
    target = db,
    write  = deny("EIO"),
)

fault_scenario("order_outbox_atomic",
    scenario = place_order,
    faults   = torn_write,
    expect   = lambda r: assert_true(r.status >= 500,
        "if the pair can't complete, the request must not claim success"),
)

Then the invariant over the real database - checked after restart, because “looks fine until restart” is this class’s signature:

def no_orphan_orders():
    rows = db.main.query(sql = """
        SELECT count(*) FROM orders o
        LEFT JOIN outbox e ON e.order_id = o.id
        WHERE e.id IS NULL
    """)
    assert_eq(rows[0][0], 0, "every order has its outbox event - no orphans")

What to assert

  • The pair is atomic: both rows or neither - verified by querying the real database, not by trusting the response.
  • The failure is loud: a 5xx, not a 201 for half-written state.
  • After restart, recovery logic actually reconciles whatever partial state the crash left.

Which level

Syscall - this is the one class only the syscall level can produce, because the failure must split a sequence the application believes is atomic. write=deny("EIO"), fsync=deny("EIO") for the WAL variant. See Choosing Fault Levels.

Learn the pieces

Edit this page on GitHub