Choosing Fault Levels: Start from the Bug Class

You don’t choose a mechanism - you choose a bug to close. Decide which of the six bug classes you’re testing for, and the fault level follows. This guide gives the mapping first, then the mechanics of each level.

Which bug are you closing?

The rest of this guide explains what each level can and cannot do, so you can refine these defaults.

Two levels, one API

# Syscall level: affects ALL writes by the db process
fault(db, write=deny("EIO"), run=scenario)

# Protocol level: affects only INSERT queries to the orders table
fault(db.pg, error(query="INSERT INTO orders*"), run=scenario)

Same fault() builtin. The first argument determines the level:

• Service (db) → syscall level
• Interface reference (db.pg) → protocol level

When to use syscall faults

Syscall faults simulate infrastructure failures - the kind that affect everything a service does, not just specific operations.

Strengths:

• Works on ANY binary - no protocol support needed
• Catches unexpected write paths (logging, temp files, metrics)
• Simulates real infrastructure failures accurately
• Simple: one line tests a broad category

Weaknesses:

• Coarse: write=deny("EIO") blocks stdout, TCP, files - everything
• Can’t target specific queries, paths, or commands
• May break service health (can’t respond to healthchecks under write fault)

Best for: “is the infrastructure broken?” questions.

When to use protocol faults

Protocol faults simulate application-level failures - one operation fails while the rest of the service works normally.

Strengths:

• Precise: target specific queries, paths, commands, topics
• Realistic: real services fail at the query level, not the disk level
• Service stays healthy - healthchecks and other operations work normally
• Tests error handling for specific code paths

Weaknesses:

• Only works for supported protocols (HTTP, Postgres, Redis, Kafka, etc.)
• Proxy adds latency (usually <1ms, but measurable)
• Can’t simulate low-level failures (disk corruption, kernel panics)

Best for: “does this specific operation handle errors correctly?” questions.

Decision table

Combining both levels

The most thorough tests use both:

def test_degraded_system():
    """Upstream is rate-limited AND local disk is slow."""
    def scenario():
        # POST is blocked by the proxy (protocol fault).
        resp = api.post(path="/orders", body='...')
        assert_eq(resp.status, 429)

        # GET still works but DB is slow (syscall fault).
        resp = api.get(path="/orders/1")
        assert_true(resp.duration_ms > 400)
    
    def with_slow_db():
        fault(db, write=delay("500ms"), run=scenario)
    
    fault(api.http,
        response(method="POST", path="/orders*", status=429),
        run=with_slow_db,
    )

When to combine:

• Testing graceful degradation (some operations fail, others slow)
• Testing cascading failures (upstream error + local resource issue)
• Testing that partial failures don’t corrupt state

Progression for a new project

Work through the bug classes in order of incident frequency:

1. Unhandled dependency failure first - connect=deny + error() for every dependency of your critical flow. This is the class behind most production incidents, and one afternoon covers it.

2. Timeouts and retries next - delay() on the same dependencies. Assert latency bounds, not just status codes.

3. Then the precision classes for flows where correctness is money: non-idempotent retries (delay() past the caller’s timeout), torn writes (write=deny("EIO")), recovery (eventually() after the fault clears), bad responses (response() rewriting).

Most projects get 80% of the value from step 1 alone - which is also why “which bug class has no spec yet?” is the right review question, not “which mechanism haven’t we used?”