Substrate Failure-Mode Catalog

M28 wave, unit 5. This document enumerates the failure modes the substrate is expected to handle correctly, along with their trigger, expected behaviour, and a pointer to the regression test that pins the behaviour. The test suite lives at tests/integration/failure_modes/test_failure_modes.py and runs on every PR.

The goal is not exhaustive coverage of every code path — it’s a regression net for the classes of failure that have shipped a real production outage or that an operator would page on. New failure modes should be added here AND covered by a test in the same PR.

---

Why this catalog exists

Three forces push us to keep this catalog disciplined:

1. Production downtime is concentrated in a small set of failure classes. Schema drift, expired credentials, DB pool collapse, and budget runaway account for the majority of substrate incidents in the M13–M27 window. A targeted regression net is cheaper to maintain than a broad chaos-engineering rig.

2. Behaviour under failure is the contract. Clients dispatch on error envelopes (error, message, retryable). A change to the failure envelope is as breaking as a change to the success payload, and our public clients (Prism /admin, Cloudflare worker, external uptime probes) will fail silently if we drift.

3. Operators read these tests as documentation. When a 503 shows up at 3am, the on-call’s first move is to grep for the message slug — having a single suite that pairs trigger → expected behaviour → test reference shortens the time to diagnosis.

---

Failure mode index

#	Failure	Trigger	Expected behaviour	Test
1	Schema partial	Deploy ran ahead of the migration job; one or more *.sql files in scidex_substrate.migrations are not in substrate_schema_migrations	GET /v1/readiness → 503 status="schema_partial". schema_drift lists the missing filenames (capped at 20); schema_drift_count carries the full delta. Operators MUST page on this — the next write to a missing table returns 500.	test_schema_partial_readiness
2	DB unreachable	pgBouncer / Postgres pool unreachable, network partition, or graceful shutdown in progress	GET /v1/readiness → 503 status="unavailable", retryable=true, retry_after_ms=5000, details.exception_class=<class> for triage. NO schema_drift field (false-confidence guard).	test_db_unreachable_readiness
3	Budget cap exhausted	AnthropicLLM.spent_usd + estimated_cost > cap_usd for the next call	Raises BudgetExceeded before any provider request fires. Carries cap_usd, spent_usd, attempted_usd. Class name contains "Budget" so the runner’s class-name sniff routes to failure_reason='budget'. spent_usd is NOT incremented on a refused call.	test_budget_cap_exhausted
4	Expired / revoked auth key	The bearer key’s row has revoked_at IS NOT NULL OR expires_at <= NOW()	ApiKeyAuthMiddleware → 401 error="unauthorized", message="invalid or revoked api key", retryable=false. The middleware DID consult auth_keys (vs short-circuiting on a missing header).	test_expired_auth_key
5	Concurrent transfer / double-spend	Two simultaneous transfers from the same wallet whose sum exceeds the available balance	Exactly one transfer succeeds. The other raises ValueError containing "insufficient" or "underflow". Final sender balance is non-negative. Net wallet delta across all legs = 0 (no value created out of thin air).	test_concurrent_transfer_no_double_spend
6	Malformed envelope (body hash)	X-Signed-Envelope header whose signed_body_hash doesn’t match the canonicalized payload (attacker mutated the payload after the signer hashed it)	ApiKeyAuthMiddleware → 401 error="unauthorized", message contains body_hash_mismatch slug verbatim, retryable=false. Clients dispatch on the slug.	test_malformed_envelope_rejected
7	Debate cancellation mid-round	asyncio.CancelledError fires while run_one_session is inside an LLM call (rolling deploy, OOM kill, budget tripwire)	The session row is rolled back to status='queued' (NOT left at running, NOT marked failed). failure_reason is None. CancelledError is re-raised so the supervisor sees the cancel cleanly. Another runner can claim the work on the next tick.	test_debate_cancellation_cleans_up

---

Per-failure detail

1. Schema partial — /v1/readiness reports schema_partial

Trigger. The deployment pipeline shipped a new substrate version whose scidex_substrate/migrations/ package contains a *.sql file that has not yet been applied to the live database. The pod boots cleanly (/readyz is happy because in-process migrations report applied=0, failed=0), but the next request that targets a missing table returns a 500. This is the silent-DOWN pattern docs/runbooks/schema_drift.md exists to detect.

Expected behaviour. GET /v1/readiness is public (no auth required) and:

• Computes expected = {*.sql in scidex_substrate.migrations}

• Reads applied = SELECT filename FROM substrate_schema_migrations

• If expected ⊋ applied, returns 503 with:

  {
    "status": "schema_partial",
    "schema_applied": <count>,
    "schema_expected": <count>,
    "schema_drift_count": <total missing>,
    "schema_drift": [<first 20 filenames, sorted>],
    "as_of": "<ISO timestamp>"
  }
  

Test. test_schema_partial_readiness stubs _expected_migration_files to return 3 names and the substrate_schema_migrations cursor to return 2. Asserts 503, schema_partial, and the missing filename in the drift list.

2. DB unreachable — /v1/readiness reports unavailable

Trigger. _database.acquire() raises. Common causes: pgBouncer is down, the Postgres primary is restarting, the network to the DB host is partitioned, or the pod is in graceful shutdown and the pool has been closed.

Expected behaviour. Return 503 with:

{
  "status": "unavailable",
  "message": "DB pool not reachable; retry shortly.",
  "details": {"exception_class": "<class>"},
  "retryable": true,
  "retry_after_ms": 5000,
  "as_of": "<ISO timestamp>"
}

Critically: no schema_drift field. We don’t know what’s in the DB, and surfacing a partial drift list when the DB itself is down would be a false-confidence signal.

Test. test_db_unreachable_readiness patches _database.acquire with a context manager that raises ConnectionError. Asserts the envelope, retryable=true, exception_class="ConnectionError", and the absence of schema_drift.

3. Budget cap exhausted — BudgetExceeded before network call

Trigger. AnthropicLLM.spent_usd plus the pre-call cost estimate would exceed cap_usd. This happens when:

• A long-running debate session blew past its per-call budget.

• The per-call estimator under-counts (the model produced an unexpectedly long response).

• An operator set SCIDEX_BUDGET_CAP_USD lower than the session’s natural cost.

Expected behaviour. Raise BudgetExceeded synchronously, before the network call. The exception:

• Subclasses RuntimeError.

• Class name contains the literal substring "Budget", which runner.run_one_session sniffs to route the session to failure_reason='budget' (vs the generic llm_error bucket).

• Carries cap_usd, spent_usd, attempted_usd so the audit log can distinguish “narrowly over the cap” from “10x over the cap”.

Invariant. spent_usd is NOT incremented on a refused call — the accountant rule is “no spend without a response”.

Test. test_budget_cap_exhausted constructs an AnthropicLLM with cap_usd=0.01, manually sets spent_usd=0.5 (already over), and installs a sentinel _client whose messages.create raises if it’s ever called. Asserts BudgetExceeded is raised, the three fields are populated, the sentinel was never touched, and spent_usd is unchanged.

4. Expired / revoked auth key — 401 invalid or revoked api key

Trigger. An operator rotated keys after a leak, the auth_key_expiry_sweeper revoked an aged key, or the key’s expires_at is in the past.

Expected behaviour. The lookup SQL filters revoked_at IS NULL AND expires_at > NOW() so a revoked/expired row simply doesn’t match. The middleware sees None from _lookup_active_key and returns:

{
  "error": "unauthorized",
  "message": "invalid or revoked api key",
  "details": {},
  "retryable": false
}

Why retryable: false. More retries don’t make a revoked key un-revoked. Clients must rotate the key (or page the operator), not back off and retry.

Test. test_expired_auth_key queues None to the FakeConn (simulating the WHERE-clause miss), fires a POST /v1/echo with a fake bearer token, and asserts the 401 envelope. Additionally asserts the middleware DID issue a query against auth_keys — guarding against accidental short-circuits where auth is bypassed without ever consulting the DB.

5. Concurrent transfer / double-spend — exactly one succeeds

Trigger. Two simultaneous transfer calls from the same wallet that, taken together, would exceed the available balance. Common causes:

• A misbehaving client fires the same transfer twice without an idempotency key.

• A retry policy fires a duplicate before the original’s response lands.

• An attacker deliberately races to spend the same funds twice.

Expected behaviour. Exactly one transfer succeeds. The losing side raises ValueError whose message contains "insufficient" (the apply_debit underflow guard path) or "underflow" (the transfer pre-check path). Final state:

• Sender balance is non-negative.

• Recipient holds exactly the successful transfer amount.

• Net delta across the whole system = 0 (no value minted).

Test. test_concurrent_transfer_no_double_spend seeds Alice with 30,000 minor units, fires two 20,000 transfers via asyncio.gather, and asserts exactly-one-succeeds, the loser raises ValueError, and the final balances are 10,000 / 20,000 / 0 (Alice / Bob / system-delta).

Production note. Real Postgres enforces this via the CHECK constraint balance_minor_units >= 0 on agent_wallets; the FakeDb mirrors that check so the test exercises the same check-then-act pattern under the same SQL surface.

6. Malformed envelope — 401 body_hash_mismatch

Trigger. An attacker (or a buggy proxy) mutated the payload of a signed envelope after the signer committed the hash, hoping the verifier wouldn’t recompute. SPEC-187’s verifier ALWAYS recomputes and rejects with a stable slug.

Expected behaviour. The middleware:

1. Parses the X-Signed-Envelope header as JSON.

2. Resolves key_id against signing_keys (returns matching pub).

3. Canonicalizes the payload and recomputes the SHA-256 hash.

4. Compares against signed_body_hash.

5. On mismatch, returns 401 with message containing body_hash_mismatch verbatim and retryable=false.

Why the slug matters. Clients dispatch on the slug:

• unknown_signing_key → re-register the key.

• signature_invalid → check signer is using the right private key.

• body_hash_mismatch → check intermediate proxies aren’t mutating the body (compression / charset normalization can break the hash).

• envelope_malformed → check the SDK is producing valid JSON.

Test. test_malformed_envelope_rejected signs a payload, mutates the payload dict after signing (leaving the signed_body_hash stale), and asserts 401 with the slug.

7. Debate cancellation — status restored to queued

Trigger. The runner is mid-round when asyncio.CancelledError fires. Common causes:

• A rolling deploy SIGTERMs the runner.

• An OOM kill terminates the process mid-call.

• A budget tripwire cancels the supervisor.

• A manual task.cancel() from an admin CLI.

Expected behaviour. The cancellation handler:

1. UPDATEs debate_sessions SET status='queued', failure_reason=NULL for the session under flight.

2. Re-raises CancelledError so the supervisor observes the cancel cleanly (not swallowed as a normal completion).

Why queued, not failed. A running row would wedge the queue — no other runner claims a running row, because that’s the “someone is actively working on this” state. A failed row would require human triage even though nothing actually failed. queued is the only state that lets a different runner pick up the work on the next tick.

Invariant. No failure_reason is carried over. If we left failure_reason='cancelled' on the row, the next runner would have to know to clear it; cleaner to leave it None.

Test. test_debate_cancellation_cleans_up constructs a slow LLM (200ms per call), starts run_one_session as an asyncio task, sleeps 100ms (so the first round is mid-flight), cancels the task, and asserts:

• The task re-raised CancelledError.

• db.sessions[sid]["status"] == "queued".

• db.sessions[sid].get("failure_reason") is None.

---

Adding a new failure mode

When you discover (or anticipate) a new failure class:

1. Add a row to the index table above with trigger / expected behaviour / test reference.

2. Add a section to “Per-failure detail” explaining the reasoning. Operators read this section at 3am.

3. Add a test to test_failure_modes.py that simulates the failure and asserts the expected behaviour. Keep it cheap (no real Postgres or Anthropic) so it runs on every PR.

4. Reference the test from any related runbook so on-call can find the contract definition.

Don’t add a failure mode without a test. The catalog without regression coverage drifts within months.

---

Running the suite

# All failure-mode tests (~2s):
PYTHONPATH=src pytest tests/integration/failure_modes/ -v

# A single failure:
PYTHONPATH=src pytest tests/integration/failure_modes/test_failure_modes.py::test_schema_partial_readiness -v

The suite is hermetic: no Postgres, no Anthropic, no env vars required beyond PYTHONPATH. CI runs it as part of the standard pytest tests/ invocation.