Reading reports & verdicts
The last stage of the verification model produces something a human reads and acts on. A fault-injection run that nobody can interpret is wasted compute. This page covers what comes out, what the verdict means, and who does what with it.
The three-valued verdict
Most test frameworks are binary: pass or fail. Faultbox is three-valued, and the third value is the important one:
| Verdict | Means | What it tells you |
|---|---|---|
| PASS | The properties held and were actually exercised | The system behaved correctly under this failure |
| FAIL | A property was violated | A real bug — here’s the event and the seed |
| INCONCLUSIVE | The property was never exercised | The fault never fired, or the system never reached the checked state |
Why INCONCLUSIVE exists: the most dangerous result in fault testing is a green check that proves nothing. If your fault targeted a syscall the service never calls, a binary framework reports PASS — and you believe you tested disk-full handling when you tested nothing. Faultbox refuses that lie. An INCONCLUSIVE leaf means “this didn’t confirm or refute the property” — it’s a prompt to fix the spec (the fault matched the wrong operation) or accept that the path genuinely can’t be reached.
A wall of PASS with hidden INCONCLUSIVE cells is the fault-testing equivalent of 100% coverage on untested code. Faultbox makes the difference visible.
What a run produces
Every faultbox test run emits a self-contained .fb bundle.
faultbox report bundle.fb turns it into one HTML file — no server, no
build, no telemetry. You email it, drop it in Slack, attach it to a ticket, or
commit it as a baseline. Opening it works offline.
The report reads top-to-bottom as a triage flow:
- Header — run ID and a PASS/FAIL pill you can screenshot as a status.
- Hero & stats — the matrix cardinality, faults actually delivered, services observed, duration. One sentence: did the system “hold up”?
- Attention — failures and warnings surfaced first, each with a
ready-to-paste
faultbox replaycommand. You don’t hunt for the red cell. - Fault matrix — the scenarios × faults grid, colour- and icon-coded (PASS/FAIL/INCONCLUSIVE/excluded). Click any cell to drill in.
- Plan — the exploration tree
that was walked: every leaf with its stable
LeafID, so the combinations that ran are auditable after the fact. - Observed coverage — per service, how many tests touched it, how many syscalls were captured, how many faulted. What actually ran, not what you declared.
- Reproducibility — Faultbox version, Go toolchain, kernel, Docker version, image digests, and a one-click “rerun this exact run.”
The drill-down
Click a failing cell and you get the reason (the assertion or property that
tripped), the faults applied (service, syscall, action, errno, hit count),
auto-generated diagnostics — patterns like FAULT_FIRED_BUT_SUCCESS (the
service silently swallowed an error), FAULT_NOT_FIRED (your rule matched the
wrong syscall), SERVICE_CRASHED — a one-line replay command, and a
swim-lane event trace with causal arrows from each fault/violation back to
its cause on the other lanes. The LeafID ties a report cell to the exact
explored leaf and its seed.
Who reads it, and what they do
The report is written for several readers at once:
| Reader | When | What they do with it |
|---|---|---|
| The engineer who wrote the spec | While building the feature | Reads the FAIL drill-down, sees the missing retry/timeout/rollback, fixes the code, re-runs |
| The PR reviewer | At review time | Opens the committed report.html; confirms the fault matrix covers the new dependency and the cells are green-for-the-right-reason (not INCONCLUSIVE) |
| On-call / incident responder | After an incident | Adds a fault assumption that reproduces the outage; the report becomes proof the fix holds and a permanent regression guard |
| Security / compliance (higher determinism levels) | Audits | Reads the observed-coverage and (at L4) the declared-I/O surface as a deliverable: “what does this binary actually talk to?” |
| An LLM agent | Authoring or triaging specs | Consumes plan.json and the structured bundle to propose missing faults (faultbox plan --suggest) or interpret a failure |
The cardinal rule: fix the code, not the cell
When a matrix cell fails, the instinct is to loosen the override until it
passes. Resist it. A failing cell is the tool doing its job — it found a
failure mode the system handles wrong. Adjusting the expectation hides the bug;
the next reader sees green and trusts a system that will page them at 3 a.m.
The correct loop:
FAIL → read the drill-down → understand the missing failure handling
→ fix the service → re-run → green for the right reasonThe only legitimate reasons to change the spec instead of the code: the expectation was genuinely wrong, or the cell is INCONCLUSIVE because the fault matched the wrong operation (then fix the fault, not the assertion).
→ Reference: Reports · Concept: Determinism & reproducibility
You’ve reached the end of the spine. The full story:
Describe your system → generate a fault matrix → explore the state space deterministically → check invariants and temporal properties → read a verdict you can trust and reproduce.
From here, do it with the Guides or learn it hands-on with the Tutorial.