/ docs
On this page

Releases

Faultbox release history. Every tagged version ships with full release notes on GitHub and a matching entry in CHANGELOG.md.

For the next version in flight, see the Issues board filtered by version label (v0.16.x, etc.).

Current

  • v0.18.0 — 2026-08-14 — Field-report fixes from the first onboarding of a large production Go service (Uber Fx, ~30 gateways, MySQL + Redis + Kafka). Seven reported issues, plus one found while building the regression corpus for the first of them. The headline: two of the reports - a host-binary SUT losing every outbound socket, and the proxy data plane freezing - were one defect, and it was not the proxy. SECCOMP_IOCTL_NOTIF_RECV returns ENOENT when a notification has already been discarded, which says nothing about the listener fd; the classifier matched it anyway by substring-searching the errno text, so the notification loop returned nil on the first one and stopped supervising for good - silently, because a nil error reads as a clean shutdown. The child kept its seccomp filter with nobody to answer it, so every intercepted syscall blocked forever. That accounts for reads and new dials both dying, for it never recovering, and for a busy connection pool always losing while a quiet client survived. A loop that ends while its target is alive now fails the test loudly, outranking every other verdict. A fault written the normal way now installs a filter: multi-line calls and spaces around = matched neither of the literal substrings the scan looked for, so the fault silently did nothing - and where the scan still cannot see a fault, the runtime now refuses it (FAULT_NOT_FILTERABLE) instead of installing something inert, so a fault either fires or the run fails saying why. mock_service() works from a containerized SUT (it bound host loopback and was injected as localhost:<port>, which resolves to the SUT itself). The Docker network survives the whole run - it was destroyed after each test and never recreated, so from test 2 every container landed on the default bridge with no name resolution. Timeouts are no longer blamed on faults that were not there, with TIMEOUT_NO_FAULT_FIRED and TIMEOUT_NO_FAULTS covering what TIMEOUT_DURING_FAULT used to absorb. replay loads specs in a subdirectory, and the Replay: hint it prints is a command replay accepts. Packet faults reach a containerized consumer - the gateway address was allocated behind a proxy-fault gate that packet faults are invisible to. Also: grpc.call() completed its first round trip against a real server (it handed []byte to grpc-go’s proto codec and failed client-side before reaching the wire), and all 13 step protocols now have a real-server spec. See CHANGELOG.md.

v0.17.x — Agent-first surface

  • v0.17.0 — 2026-07-30 — faultbox check and diagnostics for suites that cannot fail (RFC-052 gaps 1, 2 and 8). faultbox check spec.star validates a spec without launching a process, pulling an image, or needing Docker - milliseconds against the tens of seconds a run costs. The runtime could always do this; it simply was not exposed, so the only way to learn a spec was malformed was to run it. Findings carry machine-readable codes and a suggested next move, exit 0 for clean or warnings-only and 2 for errors, and the same code backs the MCP tool check_spec - a check that behaved differently through MCP would give an agent a wrong model of the tool. Two new diagnostics come from an uncomfortable finding: a CI spec exercised a broken Postgres client on every pull request for three releases and passed, because its only assertion was that a query fails under an injected fault - which a client that cannot connect at all satisfies identically. Its own comment stated the intent, and the care is what hid the bug. NO_POSITIVE_CONTROL fires when an interface is stepped but no test ever asserts a step on it succeeds; it is suite-level, which is what makes it new - a single fault-injection test asserting failure is correct, but a suite where that is the only assertion an interface receives proves nothing, and no per-test lint sees it. TEST_NO_ASSERTIONS fires when a test passed having evaluated nothing. Also ships the deprecation removals promised for v0.14.0 and never made. See CHANGELOG.md.

v0.16.x — Filesystem observation

  • v0.16.1 — 2026-07-30 — The protocol audit: eleven fixes, ten of them pre-existing. v0.16.0 fixed two Postgres bugs that survived because no spec had ever asserted on the result of a database step; this release repeats that audit across the other twelve protocols - one spec per protocol, a real server, every result checked. ready() could never succeed for eight protocols (a v0.16.0 regression - the check resolved to redis://host:port and eight plugins dialled that string verbatim, burning the whole timeout; Redis went from a 60 s failure to a 407 ms pass). MySQL steps could not authenticate or select a database, and once the credential fix let a step reach the command phase for the first time it exposed a second bug: the MySQL proxy hung forever on every result set, and because Stop() waited unbounded on the connection WaitGroup, one stuck handler hung the entire run after the test body had finished. All twelve proxies now bound teardown at ProxyStopTimeout (5 s) and emit proxy_stop_timeout rather than hanging silently. Also fixed: args= was accepted by the spec loader and silently dropped for container services; dict and list step arguments were mangled (every int, float, bool and nested value inside a dict became "", self-consistently enough that round trips looked correct); the NATS proxy corrupted every line it forwarded by mixing bufio.Scanner line-stripping with \n-only writes against a CRLF protocol; NATS publish reported success without confirming delivery; ready() made a single protocol attempt for MongoDB and Cassandra at the moment the server is least likely to be up; and every container leaked its anonymous volume - measured at 290 orphaned volumes and 18.7 GB on the dev VM, which filled a 30 GB disk and surfaced as a flaky test three specs later. New: credentials read from env= for MySQL, Redis, MongoDB, ClickHouse, Cassandra and NATS, resolved in one place so healthchecks and steps authenticate identically. Docs: faultbox setup-trace added to the CLI reference, feature-manifest rows for everything in v0.14.1 and v0.16.0, all nine protocol pages moved from tcp() to ready(), new spec-patterns Pattern 0: assert on every step (the habit both credential bugs needed to survive), and 36 broken internal links fixed. Ten of thirteen step protocols now have real-server coverage; http2, udp and grpc remain unit-tested only. Six consecutive clean go test -race ./... full-suite runs. See CHANGELOG.md.

  • v0.16.0 — 2026-07-29 — Filesystem observation (RFC-056), and two bugs found by being the first thing to assert on a database. New watch(service, files=, ops=, run=) reports a service’s file I/O with resolved paths. Faultbox could already count write calls but could not reliably say which file - a syscall carries a descriptor, and resolving it meant reading /proc out of band, racing the SUT. This closes fs-unmediated, a determinism category that had emitted no events since RFC-040, so file I/O outside declared paths was silently undetected. Built in v0.14.0 and withdrawn before release: runsc trace create instruments only tasks created after it attaches, so a network-driven query against a running Postgres produced 2 trace points where the same SQL from a freshly spawned process produced 1054 - a watch that observes nothing still runs, and every assertion under it still passes. --pod-init-config installs the session at sandbox boot instead: 236 points on that same workload. Completeness is enforced, not assumed - the canonical assertion is negative (“never wrote outside its data directory”) and is only as strong as the trace behind it, so a run fails when no sandbox connected, when points were dropped during the window, when points matched no launched service, or on a decode error. read, close and connect are opt-in via setup-trace --with-read: on a read-heavy workload, enabling reads took a run from 25,015 points and zero drops to 48,576 and 1,488, which under that rule fails the test. New faultbox setup-trace does the one-time host registration (the flag lives in daemon.json, not per container) - idempotent, reports every change and what it left alone, and prints the Docker restart rather than performing it, because that restart stops every container on the host. New ready(timeout=), protocol-aware readiness: tcp() asks whether something is listening, which for a container is true the moment Docker’s port proxy binds - measured at 0 ms against a Postgres needing ~10 s. ready() asks the service instead, via its interface’s protocol plugin and declared credentials (for Postgres, a real SELECT 1 retried to the timeout); on the RFC-056 corpus it replaced a hand-tuned 25-second sleep() with a ~2.4 s check that is correct by construction. Fixed: Postgres steps could never authenticate - the connection string carried no credentials (lib/pq fell back to the OS user, root under sudo), and the auth handshake was relayed server→client only, so SCRAM-SHA-256 (the postgres:14+ default), MD5 and cleartext all deadlocked until the client’s 60-second read deadline. Observation only: a trace point fires after the syscall, and gVisor has no fsync point, so write ordering is provable and durability is not - ops=["fsync"] is rejected rather than quietly returning nothing. New tutorial chapter: Watching the filesystem. See CHANGELOG.md.

v0.15.x — Contract-driven clients

  • v0.15.0 — 2026-07-29 — Contract-driven clients (RFC-055). Faultbox already read your OpenAPI and protobuf contracts to build the dependencies you can’t run; this release reads them from the other side, to build the callers that drive your service - and makes each caller a named actor in the trace rather than one anonymous test lane. New client() turns an OpenAPI 3.x document or a protobuf FileDescriptorSet into a named caller bound to a service interface, with its operations generated as callable attributes: mobile = client("mobile-app", target = orders.public, openapi = "./orders.yaml", validate = "response") then mobile.get_order(order_id = 42) - no path, no verb, no field names. The half that isn’t ergonomics: with the contract loaded caller-side, “under fault, the service still returned what it published” becomes a one-kwarg check. validate="response" checks each response against the schema declared for its status code, records the verdict on resp.contract_ok / resp.contract_error, and emits a contract_violation event - it deliberately does not raise, because a contract violation under fault is usually the finding, not a harness error, and an undeclared status code counts as a violation, which is how the undocumented degraded path surfaces. Clients are first-class trace actors: calls emit client_call / client_return on the client’s own swim lane with its own vector-clock participant, so three named callers against one API render as three lanes instead of one anonymous driver, and those events work as temporal anchors with no new matcher syntax (match.event(type="client_return", client="gRPC-Courier", success="false")). Clients compose with everything - calls dial through the same proxy resolution as step methods, so fault(iface, ...) applies unchanged, and TLS (RFC-038) and remote targets (RFC-036) are inherited from the interface; they are trace actors, not processes, so they take no seccomp filter and are never a fault target. New faultbox inspect --clients <spec.star> prints each client’s generated operation table. interface(name, protocol, port, spec=) is now read - the kwarg had been parsed and stored since early versions with nothing consuming it, and a client() declaring no contract of its own now inherits it. Fixed: mock_service(openapi=) and mock_service(descriptors=) resolved relative paths against the process working directory rather than the spec’s own directory, so "./api.yaml" only loaded when faultbox happened to run from the spec’s directory; and service() / client() now reject a name already taken by the other, since sharing one silently folded two participants into a single lane and vector clock. v1 is unary-only, stateless per call, and does not synthesize request data. New tutorial chapter: Contract-Driven Clients; new guide: Running in a k8s dev namespace. See CHANGELOG.md.

v0.14.x — Packet-level network faults

  • v0.14.1 — 2026-07-29 — Searching fault timing, and fixing what that search exposed. v0.14.0 shipped packet faults and a Raft harness that could express a partition but not vary when it lands. Closing that gap took one new primitive and turned up four defects, one of them a documented fault that silently did something else. New sleep(duration, clock="wall") - a wall-clock wait indifferent to event traffic. Faultbox had two ways to wait and both were conditional on the SUT: await_stable() returns on quiescence, await_event() on a matching event, and neither can hold a fault open for a fixed time. await_stable fails at it in exactly the situation the wait is for - an active fault emits the events that prevent quiescence: measured on a 3-node hashicorp/raft cluster under partition, 6681 events in three minutes with a longest quiet gap of 338 ms, so a fault-timing search that should have run 18 configurations ran 6 and hung on 12, spending 36 minutes to produce no signal. With sleep() the same search runs all 18 in about two minutes. New bandwidth(rate, dir=, queue=) and mtu(size) - the two link shapers deferred from v0.14.0. They take no matcher: a packet_* rule says what happens to packets that look a certain way, a shaper says what kind of link this is. A bare rate="1000000" is rejected because it could be bits or bytes and guessing would be a silent factor-of-eight error in the one number the fault depends on; queue= bounds the backlog in time, so the shaper drops when saturated the way a real bottleneck does. Fixed: packet_delay(dir="s2c") was a silent drop, not a delay - egress built a batch list and freed it on return, so a packet released later by the defer queue’s timer was appended to a list nobody would ever write; packet_reorder on egress had the same fate, and both existing delay tests drove the ingress path, so nothing caught it. The packet gateway’s TUN device no longer bricks a host when a run is interrupted - it was the shared, persistent constant faultbox0, so concurrent runs collided and any run that died without teardown failed every later packet-fault run with device or resource busy, recoverable only by sudo ip link delete faultbox0 and documented nowhere; devices are now per-process (fbox<pid>), SIGTERM is handled alongside SIGINT, and every run sweeps orphaned devices whose owning process is gone. An interrupted suite no longer invents verdicts for tests it never ran - RunAll had no cancellation check, so Ctrl-C left each remaining test to start, inherit a dead context and be recorded INCONCLUSIVE; the new Aborted counter is kept distinct, since “indeterminate” and “never started” call for different responses. faultbox plan was blind to choose() - the construct most likely to blow a budget - so a spec that runs 24 leaves reported Total: 2 plan instances, under-reporting the --check-cost --max-instances N gate by 12×; axes are now read statically from the spec AST, with computed option lists reported as (computed — size unknown) and the total as at least N. Inconclusive verdicts now print their reason, which had been carried on TestResult.Reason and discarded by the summary the whole time. See CHANGELOG.md.

  • v0.14.0 — 2026-07-28 — Packet-level network faults (RFC-054). Faultbox mediated at two layers - individual syscalls and parsed L7 protocol messages - with nothing in between. A packet was not an object anywhere in the codebase, so “drop this TCP segment”, “delay every ACK from the server” and “advertise a zero receive window” were inexpressible. This adds that layer using gVisor’s userspace TCP/IP stack (gvisor.dev/gvisor/pkg/tcpip) as a plain Go dependency - no fork, no runsc. The capability that motivates it: a dropped packet sends no RST. The existing drop() closes the connection, and well-written clients handle ECONNRESET correctly on the first try; a socket stuck in ESTABLISHED writing into a void until a keepalive fires - the failure that actually takes production down - was unreachable. It is now one line. Eight new faults (packet_drop, packet_delay, packet_reorder, packet_duplicate, packet_corrupt, packet_reset, packet_window, packet_pass), opt in with determinism(runtime = "gvisor") on Linux with CAP_NET_ADMIN. The packet matcher takes dir, proto, flags ("PSH,ACK", "!RST"), port, len/len_gt/len_lt, payload_prefix, payload_contains, plus the same nth/after/every occurrence selectors and probability/max_fires/mode semantics as syscall faults, and a where= lambda escape hatch over a read-only Packet value. Peer-mesh topologies now work: gateway address allocation was gated on a proxy address existing, and a mesh is a cycle, so for at least one link the proxy was always absent when the consumer’s env was built - that link went unmediated. Measured on a 3-node hashicorp/raft cluster, a leader cut off from both followers committed 88 applies before the fix and 0 after. source= now reaches rule installation (it was parsed, stored and traced, then dropped, so the documented fault(kafka.main, source=worker, drop(...)) fired for every consumer), and partition() is rebuilt on the packet gateway with direction= and partition_start() / partition_stop() - the old implementation denied connect(), which only blocks connection setup, so against any service that pools connections it silently did nothing. Also fixed: ** path globs now cross directories (op(path = "/data/**") matched nothing, so a rule targeting a database that nests its files never fired - no fault, no diagnostic, test passed), and events(where = ...) was blind to most event types, filtering to four families before invoking the lambda so the documented e.type == "proxy" predicate could never have worked. New tutorial chapter: Packet-Level Faults. Known limits at this release: watch() deferred to v0.16.0, bandwidth()/mtu() deferred to v0.14.1, packet faults act below TLS, and they do not yet participate in fault_matrix() fan-out. See CHANGELOG.md.

v0.13.x — Verification model

  • v0.13.3 — 2026-07-14 — Bug fixes + stricter spec loading, from the July 2026 documentation audit. Five silent no-ops are fixed: drop(query=/command=) was meant to scope a drop to a SQL statement or Redis command but silently dropped all traffic (an empty pattern matched everything) — now it matches; Kafka duplicate(topic=) was a dead rule — now it re-sends the produce so the consumer sees the message twice while the producer still gets one ack; proxy events dropped the action/protocol fields, so assert_eventually(...action == "error") could never fire — now they carry them. Spec loading is stricter: deny() rejects errno names outside the supported table (widened to include the documented ENOSYS / EDEADLK / ELOOP / EDQUOT / ENOLCK), and service() plus every proxy fault builtin (response/error/drop/delay/duplicate) reject unknown keyword arguments with migration hints instead of silently ignoring them — specs relying on silently-dropped kwargs (cmd=, http=, tcp=, name=) must switch to the documented forms. subject= is now an accepted topic= alias for NATS matchers, and proxy_conn_close / proxy_stall events carry the protocol field. Full host suite + go test -race + go vet green; Lima sweep 21/21 PASS. See CHANGELOG.md.

  • v0.13.2 — 2026-07-06 — Finite-trace verdict semantics (RFC-049) + the documentation and site restructure around the six bug classes. Temporal verdicts are now grounded in finite-trace logic (D4 of the RFC-047 research agenda). A test() that ends by hitting its timeout is now always INCONCLUSIVE, never PASS — even if every eventually() it declared was satisfied before the deadline — because the body never reached a declared completion (natural return or terminate_when=), so the run is a truncated prefix and a green verdict would over-claim. An unbounded always(p) (no between=) under timeout is INCONCLUSIVE for the same reason (LTL₃ prefix); at natural completion or terminate_when= it stays a definitive PASS (LTL_f end-of-trace). Bounded always(p, between=(a,b)) is unchanged. New vacuous_property warning event: when an always(p, between=) start anchor never fires, the window never opens and the predicate is never evaluated — the verdict stays PASS (the window may be legitimately untriggered, e.g. between=(error, recovery) in a run with no error) but the warning surfaces a typo’d or misnamed anchor instead of hiding as a silent green. Legacy synchronous def test_*() functions are unaffected. Full host suite + go vet green; Lima sweep 21/21 PASS across the 6 integration spec suites. See CHANGELOG.md.

  • v0.13.1 — 2026-06-18 — Fixes from the first field evaluation of v0.13.0 against a real service (truck-api). --test matching no tests now exits non-zero instead of reading as a green suite in CI, and a collapsed fault_matrix name (test_matrix_create_order) selects every expanded cell under it. An unstartable target now fails fast with the named path (exec <path>: no such file or directory) instead of a 60-second healthcheck timeout with a misleading exit_code=0 — both the binary-mode launch and the container shim carry the path, and the healthcheck races against session exit. step_recv trace events record the response body (2 KB) on non-2xx HTTP responses, so debugging a 400/500 reads straight off the trace or report. New make install-lima cross-compiles and installs both faultbox and faultbox-shim into the Lima VM’s /usr/local/bin (container mode needs both side by side); a stale hardcoded shim path and an incorrect VM_PROJECT were cleaned up. The monitor() signature change from v0.13.0 keeps its hard error (accepted breaking change — no real users to migrate yet). Full host suite + go vet green; testops corpus re-verified on the Lima VM (real seccomp + Docker). See CHANGELOG.md.

  • v0.13.0 — 2026-05-29 — Five RFCs ship together as one coordinated epic: determinism levels (RFC-040), temporal properties (RFC-041), exploration plan (RFC-042), non-deterministic operators (RFC-043), and spec language simplification (RFC-044). The new determinism() top-level builtin declares the spec’s reproducibility contract — defaults to L1 (mediated-event determinism) with strict mode on, so any unmediated I/O (clock reads, RNG, DNS to a non-Faultbox resolver, connect() to an undeclared address) fails the test at the first untolerated leak. RFC-041 adds five temporal primitives — eventually(p), always(p, between=), await_event(matcher), await_stable(quiescence_window=), and a rewritten monitor(name, on=, state_init=, update=, check=) — plus a declarative test(name, body=, expect=, timeout=, terminate_when=) builtin and the PASS/FAIL/INCONCLUSIVE three-valued verdict. RFC-042 introduces faultbox plan (static plan-tree analysis without launching services), plan.json in every bundle, the report’s Plan tab, --coverage / --suggest / --check-cost --max-instances N, and the body-re-execution engine that turns named choose("name", [opts]) axes, syscall-level probability fan-out (max_fires=/mode="exhaustive"), and parallel(..., interleavings=) orderings into multi-leaf test executions with stable LeafID attribution flowing through TestResult → bundle manifest → HTML report. RFC-043 ships the four operators (choose, nondet, halt, assume) with per-leaf assume= evaluation and an AST denylist sandbox. RFC-044 withdraws RFC-013 (param() superseded by choose()) and RFC-002 (domain() — service()/interface() proved sufficient); unifies the three plan-tree fan-out axis kinds under one NonDeterministicChoice interface; collapses the event-source and decoder surfaces under observe.stdout/observe.stderr and decoder("name", ...); deprecates faultbox generate in favor of faultbox plan --suggest. Two new tutorial chapters (Part 4 — Safety & Verification) walk the operator and fan-out vocabulary end-to-end. Full repo go test -race ./... green; Lima sweep 21/21 PASS across the same 6 integration spec suites as v0.12.29.

v0.12.x — TLS-aware proxy

  • v0.12.29 — 2026-05-02 — Remote services (RFC-036). New service(remote=...) kwarg points Faultbox at an externally-running endpoint — typically a real pod in a customer’s k8s dev cluster — without launching a process. The proxy datapath from RFC-024 dials the remote upstream and the SUT reaches it through the proxy unchanged, so every protocol-level fault (response(), error(), slow(), gRPC method targeting, SQL matchers) keeps working. Process-level kwargs (seed=, reset=, reuse=, volumes=, ports=, args=, seccomp=, observe=, ops=) and syscall-level faults are rejected at spec load with explicit error messages naming the offending kwarg and pointing at protocol faults or mock_service(). Composes with RFC-038 tls=tls_cert(...) for TLS-required upstreams (the auto-generated proxy cert covers 127.0.0.1 so SUT-side verification works against the env-rewritten loopback addr). New typed remotes({"iface": "host:port"}) value for services whose interfaces live on different hosts. New @faultbox/discovery/k8s.star stdlib helper exposing k8s.service, k8s.endpoint, and k8s.local — pure string sugar over <name>.<namespace>.svc.cluster.local, no runtime k8s client. The .fb bundle’s env.json records every (service, interface, host, protocol) tuple from a remote-using run; faultbox replay warns when replaying such a bundle and points at RFC-037 (the open companion design RFC for the offline-replay determinism story). Cluster connectivity is the user’s responsibility — Telepresence connect, kubectl port-forward, in-cluster execution, or VPN — documented in the new Connectivity guide. 49 new tests across spec-load validation (32), runtime/proxy lifecycle (10 incl. TLS×remote interop), bundle round-trip (2), replay warning (2), and string-grep doc gates (3). Full repo go test ./... green; go vet ./... clean; make demo-container Lima smoke 4/4 PASS (proxy datapath refactor confirmed non-regressive against the seccomp + Postgres + Redis path).

  • v0.12.28 — 2026-05-02 — TLS-aware proxy (RFC-038) + proxy traffic observability (RFC-034) + container fault paths (RFC-035). Twelve patch versions consolidated into one release; the per-version detail lives in CHANGELOG.md. Headline: declare interface(..., tls=tls_cert(...)) on a service interface and the proxy terminates TLS at its listener and re-establishes TLS dialing the upstream — all protocol-aware fault rules (http.error(path=...), grpc.error(method=...), kafka.drop(topic=...), redis.error(key=...)) keep firing on the plaintext between the two TLS legs. Six plugins ship migrated this release (http, http2, gRPC, Kafka, Redis, TCP); the remaining 8 (postgres, mysql, mongodb, cassandra, clickhouse, memcached, nats, amqp) are tracked in RFC-039 — declarations against unmigrated plugins emit a proxy_tls_pending event. The tls_cert(...) builtin is kwargs-only with full spec-load validation (cert/key pairing, file existence, CA PEM parse, insecure=True + ca= exclusion); empty tls_cert() auto-generates a self-signed proxy cert in memory for dev/test. Two TLS plumbing patterns landed across the 6 plugins: listener wrap-and-dial via proxy.ListenTLS(serverCfg) + proxy.Dial(ctx, target, clientCfg) (http, http2, kafka, redis, tcp) and framework credentials via grpc.Creds(credentials.NewTLS(...)) for gRPC (whose server owns its own TLS handshake). Also in this release: RFC-034 connection lifecycle observability — proxy_conn_open, proxy_conn_close with byte counts and reason classification, proxy_handshake_complete, and proxy_stall (1Hz watchdog, 5s warn / 30s extend tiers) — wired into 13 of 15 plugins (udp/grpc deferred). RFC-035 container-consumer fault paths on Linux Docker (proxy bind defaults to 0.0.0.0 so container consumers reach proxies via host.docker.internal). New stderr() event source. Container-mode observe=[stdout(...)] now works for container services (was binary-mode only). Race fix on tcp.go::handle — both io.Copy goroutines were writing closeReason concurrently. New internal/proxy/{http,grpc,kafka,redis,tcp}_tls_test.go suites (~30 tests). Full repo go test ./... -race green; Lima sweep 21/21 PASS across 6 integration spec suites.

v0.12.x — Scalable HTML Reports

  • v0.12.16 — 2026-04-30 — Report UX overhaul. Driven by inDrive Freight triage feedback on the v0.12.15.x customer report; entirely scoped to internal/report and the docs that describe it (no bundle format or spec-language changes). Causal links now follow cause, not chronology — findCausalAncestors switched from vector-clock partial order (which on real bundles routinely had only the lifecycle events with complete clocks, so spaghetti pointed at service_ready instead of proxy_fault_applied) to seq-based strict precedence, restricted to faults / violations / errored steps. Hovering an ordinary success step now draws zero lines. New timeline filter bar above every Event Trace block: three presets (Compact default, hides framework lifecycle chatter; Anchors only strips everything except cause-relevant events; All events historical default) plus free-text search across event type / headline / fields. proxy_fault_applied / proxy_fault_removed are now first-class fault markers (red, not default-blue) across markerKind/severityScore/isAnchorEvent/eventHeadline; added to Phase 3 anchorTypes so they survive downsampling. Per-test “Faults applied” section pairs proxy_fault_applied with proxy_fault_removed (one row per assumption: service · protocol · interface · assumption · seq window). Recent block in the Assertion drill-down interleaves fault events between captured step rows by seq, with a fade-and-expand cap (220px max-height + mask-image gradient + Show full assertion toggle, 320ms transition) so long Recent lists / Actual reprs stop dominating the dialog. Group-members table on folded markers (paginated 100/page, sticky header, scrollable) replaces the one-line “collapsed run” hint — runs that hide a 5xx among 99 successes are now legible. Fullscreen toggle on the test details modal (⤢ ↔ ⤡). STDOUT JSON renders as a 2-column key/value table with dot-path flattening for nested objects. Source block falls back to fault_matrix(...) for matrix-generated test names and surfaces three jump links (scenario def get_order_feed(), fault mysql_slow = fault_assumption(...), matrix call site) — was previously “Could not locate def…” for every matrix cell. Plus tooltip vertical-text fix (width:max-content + four-side viewport clamping), detail panel summary no longer truncates, folded marker click routes through markerEvBySeq so the Group-members table actually appears.

  • v0.12.15.2 — 2026-04-30 — Hotfix on top of v0.12.15.1. Customer verified the redis RESP3 fix landed clean (cold-start path green end-to-end, smoke PASS in 16.3 s) but the failure moved to the reuse path: cell 1 of the dbmatrix run passes, cells 2–18 all fail identically with error connect to db: invalid connection (go-sql-driver ErrBadConn) or read: connection reset by peer (go-redis). Root cause: Manager.EnsureProxy rooted each proxy’s Accept goroutine at the caller’s ctx. preStartProxies runs under RunTest’s per-test testCtx which cancels via defer cancel() at end of test — at end of cell 1 that cancellation took down the goroutine while the listener fd stayed bound (only Stop() closes it) and the cached m.proxies[key] entry stayed in place. Cells 2..N hit EnsureProxy → cache hit → proxy_active(reused) fired → but nobody was Accept()-ing, so the kernel completed the TCP handshake and then RST-ed. v0.12.15.2 roots the proxy’s pCtx at context.Background() so its lifetime is bound to Manager.StopAll/StopService (which already drive explicit teardown). Why this surfaced now: pre-v0.12.13 no-seccomp containers were destroyed every cell, forcing fresh proxies via the cold path; v0.12.13 fixed reuse so containers AND proxies stay alive across cells, exposing the latent ctx-rooting bug. New regression test TestManagerEnsureProxy_SurvivesCallerCtxCancel. Lima sweep 21/21 PASS.

  • v0.12.15.1 — 2026-04-29 — Hotfix on top of v0.12.15. Customer verified the MySQL fix landed clean (Finding H closed) but the failure moved one step forward: truck-api hung 6s on its first Redis Ping(). Root cause: go-redis v9 unconditionally sends HELLO 3 from initConn, which switches the server to RESP3; v0.12.15’s redis proxy readRESPRaw only knew RESP2 framing (+, -, :, $, *) and on a %N map header fell through to the default branch — returned just the header line and left the map body unread on the upstream socket. Wire-level proof: redis-cli -p $PROXY -3 PING timed out 6s, redis-cli -p 16379 -3 PING (direct) returned PONG in 8ms. v0.12.15.1 extends readRESPRaw to cover RESP3 aggregates (% map, ~ set, > push, | attribute) and scalars (_ # , ( single-line; = ! bulk-string-framed). Four new regression tests including TestRedisProxy_RESP3_HelloMap reproducing the customer’s exact map shape.

  • v0.12.15 — 2026-04-29 — Hotfix on top of v0.12.14. Customer verified v0.12.14 didn’t unblock Finding H — both caching_sha2_password and mysql_native_password --default-auth still hung 8s through the proxy. Root cause: v0.12.14’s handshake loop assumed strict client/server alternation, but caching_sha2_password fast-auth-success (taken when the user is in the server’s auth cache) emits two server-side packets back-to-back — AuthMoreData(0x01, 0x03) then OK(0x00) — with no client packet between. The proxy read the AuthMoreData, tried to read from the client, deadlocked. Customer’s seed_db populated the auth cache via direct MySQL connections, so every proxy connection hit fast-auth. v0.12.15 peeks the second byte of every AuthMoreData and treats 0x03 (fast_auth_success) as “expect another server packet” rather than “expect client reply”. New regression test guards the path.

  • v0.12.14 — 2026-04-29 — First attempt at Finding H. Loops the handshake until OK/ERR, alternating directions on AuthMoreData/AuthSwitchRequest. Fixed caching_sha2_password cold-cache full-auth (covered by TestMySQLProxy_Handshake_CachingSha2FullAuth), but missed fast-auth-success (back-to-back server packets) — superseded by v0.12.15 the same day. Also bumps step.summary preview cap from 80 → 500 chars.

  • v0.12.13 — 2026-04-28 — Hotfix on top of v0.12.12. Pre-existing bug (RFC-015 vintage): container services without seccomp filters bypassed rt.sessions registration, so stopServices’ reused set silently ignored reuse=True for proxy-only Docker upstreams — container destroyed every cell, but the proxy in proxyMgr was kept pointing at the dead host port. Fix populates rt.sessions in the no-seccomp branch and nil-guards ClearDynamicFaultRules in the reuse path.

  • v0.12.12 — 2026-04-27 — Proxy-address surface for host-binary SUT + Docker upstream (RFC-033). New iface.proxy_addr / proxy_host / proxy_port interface attributes give a working spec-language path for wiring host-binary SUTs through the fault-injection proxy — late-bound at spec-load, resolved at buildEnv time, no rsplit() games. Also emits proxy_active events in the reuse=True path so per-cell trace partitions reflect what’s wired up at cell start.

  • v0.12.11 — 2026-04-26 — Compact fold-count labels. Run-marker badges now render counts as × 3.9k / × 86k / × 4M instead of full numerals; the exact count remains in the badge’s hover tooltip. Truncates rather than rounds — × 3.9k always represents ≥ 3900 events.

  • v0.12.10 — 2026-04-26 — Spec-anchored event highlighting. The runtime tags step_send / step_recv events with fields.spec = <test_name> whenever the call originates from inside the currently-running test function (helper functions still register — the test frame is on the stack). Renderer paints these markers with a warm gold ring, prefixes the balloon / log headline with ★, bumps severity by +50 so they win their slot, and bypasses the lane fold so the user’s own calls always render individually against background traffic.

  • v0.12.9 — 2026-04-26 — Three UX polishes. Run-marker discs scale with log10(fold count) — magnitude is visible at-a-glance even when adjacent chips’ badges would otherwise overlap (count text now sits inside the disc). Drill-down “All fields” / “Vector clock” expansion state persists across pin changes. Step summaries pair the directional arrow with explicit call / reply words — → call · truck-api.get /orders, ← reply · truck-api.get /orders [500].

  • v0.12.8 — 2026-04-26 — Lane filter now folds by key (target.method.summary) before slot bucketing — 1787 identical SELECT 1 errors collapse to a single × 1787 chip instead of 50 indistinguishable red dots. Causal hover lines restored for v0.12.7’s lane routing (cross-lane detection uses laneFor; folded ancestors resolve to their containing chip via _runMembers). Type filter axis becomes click-to-add — the toolbar stays empty until the user clicks a Type cell in the table.

  • v0.12.7 — 2026-04-25 — Step events now lane on their target service: db.exec(...) lands on the db lane, truck-api.get(...) on the truck-api lane, instead of all of them piling onto the test driver lane. Event-log filter applies to the full event set (not just the first 200 loaded rows), so filtering by a service finds matches anywhere in the trace.

  • v0.12.6 — 2026-04-25 — Lane markers color by severity (failed steps + 5xx → red, 4xx → amber); slot picker prefers severity over first-anchor; Recent trail ellipsizes long lines with hover-tooltip; two-axis event-log filter (Service + Type) replaces the v0.11 single-select chip bar — click a cell in the table to filter to its value.

  • v0.12.5 — 2026-04-25 — Hard per-lane marker budget. Walks back the v0.12.2-4 dedup/window approaches (none had an upper bound on DOM node count). New filter buckets each lane’s events into 50 visual slots in seq order; each slot picks one representative (anchor > fold-key head > first event). Hard guarantee: ≤ 50 markers per lane regardless of input size — the customer’s 86874-event lane now renders 50 markers (≈ 250× DOM reduction) at uniformly-spaced visual positions.

  • v0.12.4 — 2026-04-25 — Lane filter rewritten as anchor windows + global cardinality fold (faults / violations / errored steps + ±10 around each render per-event; outside groups by (target, method, summary) and folds large buckets to a × N chip at the median rank). Customer’s noisy 80k-event test now shows ~5 distinct chips. Assertion drill-down gains a “Recent” trail snapshotting the last few step events at fail time so resp.status = 500 is visible inline.

  • v0.12.3 — 2026-04-25 — Drill-down ergonomics. Assertion block now lifts the original expression text from the spec source (assert_true(resp.status in [200, 201]) shows Expression: resp.status in [200, 201], not just Actual: False), with a clickable spec.star:42 location link. Lane dedup also keys on summary so mixed SQL doesn’t flatten into one chip. Click on a lane marker no longer scrolls the page.

  • v0.12.2 — 2026-04-25 — Step-event readability. Runtime enriches step_send/step_recv with sql/query/path/args/status_code/duration_ms/error plus a one-line summary preview; lane dedups consecutive (target, method) runs into a single × N marker (a 1500-iteration db.exec loop renders one chip instead of 3000). FAQ entry added on bundle freezing — old bundles can’t invent new fields; re-run the suite on v0.12.2 to benefit.

  • v0.12.1 — 2026-04-25 — Drill-down polish: structured assert_eq / assert_true Expected vs Actual block, services list now shown for proxy-mode runs (event-log fallback), swim-lane switched to rank-based axis with syscalls relegated to the event-log table — keeps timelines legible at 80k+ events.

  • v0.12.0 — 2026-04-25 — Report architecture redesign (RFC-031). 23 MB reports become ~150 KB by default; --full-events opt-out preserves every event. Plus panic-safe bundles, binary-digest pinning, actionable lock drift, grpc.retryable(), proxy-coverage CI gate, positioning doc.

v0.11.x — Interactive HTML Reports

  • v0.11.3 — 2026-04-25 — MySQL driver EOF noise suppressed, CHANGELOG + per-release pages land.
  • v0.11.2 — 2026-04-24 — Hotfix: gRPC proxy passthrough + fault_matrix mock-target panic. --test glob/regex. Capability matrix in README.
  • v0.11.1 — 2026-04-24 — Five-outcome matrix: expectation_violated (amber), fault_bypassed (grey, opt-in via require_faults_fire).
  • v0.11.0 — 2026-04-24 — Interactive single-file HTML reports. faultbox report <bundle.fb>. Swim-lane trace viewer.

v0.10.x — Reproducibility trio completes

  • v0.10.1 — 2026-04-23 — Assumption ProxyRules applied in fault_scenario/fault_matrix. testops corpus: Critical tier 100% green.
  • v0.10.0 — 2026-04-23 — faultbox replay <bundle>, faultbox lock + faultbox.lock (image-digest pinning, RFC-030).

v0.9.x — Reproducibility, docs, primitives

  • v0.9.9 — 2026-04-23 — JWT/JWKS mock, documentation overhaul (~1500 lines, 6 new pages).
  • v0.9.8 — 2026-04-23 — load_file()/load_yaml()/load_json(), expect_* predicates, gRPC status shorthands.
  • v0.9.7 — 2026-04-22 — .fb bundle format (RFC-025), faultbox inspect, always-on reproducibility.

Older

See the complete CHANGELOG for entries prior to v0.9.7 and the GitHub Releases page for the canonical per-version notes.

Edit this page on GitHub