Runbook — Substrate v2 Rescue

Recovery procedure for a degraded or wedged substrate process.

Source: docs/runbooks/substrate_v2_rescue.md

Substrate v2 rescue runbook

When the substrate v2 service is silently dead — systemctl is-active reports active, but port 8200 is unbound and nginx proxies /api/* to a 502 — five known-failing migrations have left the scidex_v2 schema in a partial state and _lifespan crashed mid-boot. This runbook walks an operator through detecting that exact failure mode and reconciling the schema with the two tools shipped for it:

  • scripts/ops/check_substrate_health.py (M42-6 #796) — multi-layer probe; the only tool that reliably distinguishes silent-death from honest failure.

  • scripts/ops/rescue_substrate_v2.py (M43-4 #800) — state-aware per-statement reconciler scoped to the 5 known-failing migrations (0072 / 0076 / 0078 / 0082 / 0085).

For schema drift outside these 5 files, use the broader substrate-v2-schema-reconciliation.md. Incident background lives in docs/operations/v2-session-retrospective-2026-05-14.md and docs/operations/v2-endpoint-coverage-2026-05-14.md (Migration audit section).

1. Symptom signatures — recognising the silent death

You are looking at this failure when all four of the following hold:

Signal Healthy value Silent-death value
systemctl is-active scidex-substrate-v2 active active
ss -ltn 'sport = :8200' (or curl -s 127.0.0.1:8200/health) listening / 200 nothing bound / connection refused
nginx response on prism.scidex.ai/api/... 200 / 4xx 502 Bad Gateway
substrate_schema_migrations row count 80+ < 80 (pending the 5 below)

The diagnostic giveaway is systemd-active + port-unbound. Uvicorn workers died inside _lifespan after systemd already recorded the parent as started — the unit looks healthy but no worker serves traffic.

2. Diagnosis — one command, structured verdict

Always run the health check first; it is read-only and never writes to the DB.

cd /home/ubuntu/scidex-substrate     # or wherever substrate is checked out
PYTHONPATH=src python3 scripts/ops/check_substrate_health.py --operator-mode
echo "exit=$?"

What --operator-mode adds: a tail of journalctl -u scidex-substrate-v2 so you can see the actual _lifespan traceback that killed the workers. Skip the flag if you do not have journal read access — the rest of the probe still works.

Exit-code semantics (also printed on the verdict line):

Code Verdict Meaning
0 healthy Service up, all migrations applied.
1 service-down or SILENT DEATH Unit active but port 8200 unbound, or /health not 2xx. This runbook applies.
2 schema-partial /readyz returned 503 or journal shows pending migrations. Often paired with exit-1 when the partial state crashed _lifespan outright. This runbook also applies.
3 db-unreachable Cannot SELECT from scidex_v2. Stop — fix Postgres first; this runbook will not help.

A typical silent-death report:

substrate health check — SILENT DEATH: unit active but port 8200 unbound
  exit_code: 1

  [OK  ] systemd scidex-substrate-v2: active
  [FAIL] tcp 127.0.0.1:8200: [Errno 111] Connection refused
  [FAIL] http /health: status=URLError: ConnectionRefusedError(111, ...)
  [FAIL] http /readyz: status=...
  [OK  ] db postgresql://scidex_app:***@localhost:5432/scidex_v2: reachable, public_table_count=178
  [FAIL] migrations: applied=75 on_disk=80 pending=5 latest_on_disk=0085_... latest_applied=0071_...

The pending=5 + the on-disk/applied gap names the failure: the 5 migrations listed in section 4 below.

3. Rescue procedure

The rescue script is scoped to exactly these 5 filenames:

0072_senate_proposals.sql
0076_token_ledger_v2.sql
0078_proposal_signatures.sql
0082_agent_registry_wallets.sql
0085_spec184_recovery_proposals.sql

It probes per-statement using information_schema / pg_indexes / pg_constraint, runs only the missing statements inside a savepoint, re-probes, and only then marks the migration applied. Re-running is a no-op. The script does not modify migration files or the runner.

3a. Dry-run reconcile (always do this first)

PYTHONPATH=src python3 scripts/ops/rescue_substrate_v2.py --dry-run
echo "exit=$?"

Each migration prints a per-statement plan tagged PRESENT / MISSING / skip / UNKNOWN, then a summary line:

  TOTAL: 7 statement(s) need apply, 0 applied this run, 0 error(s).

Exit code 0 with total_need_apply > 0 is the expected pre-apply state. Exit code 3 means the substrate_schema_migrations journal does not exist — boot substrate once (or use the broader reconciliation runbook) to initialise it before retrying.

3b. Apply reconcile

PYTHONPATH=src python3 scripts/ops/rescue_substrate_v2.py --apply
echo "exit=$?"

Expected output: total_applied_now == total_need_apply, total_errors == 0, exit code 0. Each successful migration prints -> recorded in substrate_schema_migrations underneath its plan. The script commits after each migration so a later failure does not roll back earlier successes.

3c. Restart the service

sudo systemctl restart scidex-substrate-v2

This clears the _lifespan crash loop and gives uvicorn a fresh chance to bind port 8200.

3d. Verify recovery

PYTHONPATH=src python3 scripts/ops/check_substrate_health.py

You want exit code 0 / healthy, with /health and /readyz both 200 and pending=0. Spot-check the endpoint that originally surfaced the incident:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8200/v1/proposals
# 200 expected (was 502 / 500 before rescue)

4. Per-migration details

What each of the 5 migrations does and why it crashed the legacy runner. State-aware reconcile handles each correctly today; this table is for the operator who wants to know what just got fixed.

File Purpose Pre-fix failure
0072_senate_proposals.sql Creates proposals + proposal_votes (SPEC-184 §21, M11-5). First v2 governance HTTP surface. UndefinedColumn "voter_agent_id" when applied via async psycopg. Same SQL applies cleanly via psql -f — the divergence is in psycopg’s async multi-statement DDL handling, fixed at the runner level by PR #698’s statement-split. Partial state on prod: some CREATE TABLEs landed before the failing statement, but the journal row was never written.
0076_token_ledger_v2.sql SPEC-178 §28 spec-compliant token_ledger_v2 (coexists with v1 token_ledger). Index creation UndefinedColumn or UndefinedTable on legacy runner. Includes a view agent_wallets that a later migration (0095) supersedes with a real table — the rescue script’s supersession rule recognises this and skips the CREATE OR REPLACE VIEW to avoid WrongObjectType.
0078_proposal_signatures.sql MultiSig overlay on senate proposals (SPEC-186 §22.3, §22.4 — M14-3). Multi-statement failure mid-file on legacy runner; partial state left the signatures table half-created.
0082_agent_registry_wallets.sql agent_registry table for SPEC-173 §6.9 wallet pointers (M15-4). Same class — partial state from mid-file failure.
0085_spec184_recovery_proposals.sql SPEC-184 §21.2 #3 — recovery proposal kind + agent_pauses (M16-2). Extends the proposals proposal_type enum via DROP CONSTRAINT … ADD CONSTRAINT …. Coupled DROP+ADD constraint pair: the legacy runner treated them as independent statements; if the DROP committed but the ADD raised, the table was left missing its enum check entirely. The rescue script detects the coupled pair (same table + constraint name) and forces both halves to run atomically.

The runner-side fix (PR #698) split the SQL into individual statements so each commits or rolls back deterministically; the rescue script exists to clean up databases that already accumulated partial state BEFORE that fix landed.

5. Rollback — if reconcile goes wrong

The reconcile runs each migration in its own transaction, so the typical “rollback” is automatic: a failure rolls back that migration without touching the others, and the journal row is not written. You can re-run --dry-run to see what landed and what did not.

If you need a stronger rollback (e.g. one of the partial applies left behavior worse than before), restore the most recent scidex_v2 PG dump. Since 2026-05-01 substrate’s PG dumps are produced by Reprise (https://github.com/SciDEX-AI/Reprise) per its per-cycle (every 5 min) schedule defined in hosts/sandbox-02.yaml — see also .reprise.yaml at the repo root. The new local path is the project-scoped /data/backups/${BACKUP_MACHINE_ID}/scidex-substrate/postgres/scidex_v2-*.sql.gz (was the flat /data/backups/${BACKUP_MACHINE_ID}/postgres/ under the retired scripts/ops/backup.sh/Orchestra cron).

ls -lt /data/backups/${BACKUP_MACHINE_ID}/scidex-substrate/postgres/scidex_v2-*.sql.gz | head -5

# Restore (operator only — DESTRUCTIVE, replaces current DB).
# Prefer `reprise restore` (see Reprise README) for the new layout;
# the gunzip|psql form below remains valid for either layout.
sudo systemctl stop scidex-substrate-v2
gunzip -c /data/backups/${BACKUP_MACHINE_ID}/scidex-substrate/postgres/scidex_v2-<ts>.sql.gz \
  | PGPASSWORD=scidex_local_dev psql -U scidex_v2_app -h localhost \
      -d scidex_v2 -v ON_ERROR_STOP=1
sudo systemctl start scidex-substrate-v2
PYTHONPATH=src python3 scripts/ops/check_substrate_health.py

If reconcile applied cleanly but the service still won’t come up, the issue is downstream — read the journalctl tail in the operator-mode health check and triage from there. Do not loop --apply; the script is deterministic.

6. Prevention

Once the rescue has shipped, the same partial state should never re-accumulate. To keep it that way:

  1. Health check after every deploy. Wire check_substrate_health.py (exit code 0 required) into the post-deploy smoke. A --json flag is available for automation.

  2. Treat /readyz 503 as deploy failure. PRs #745 + #782 made /readyz the schema-drift probe; LBs and deploy smokes should probe it instead of /health (which stays 200 on partial schema).

  3. CI gate: fresh-DB apply. A CI job that applies every migration against a fresh scidex_v2_ci DB catches the next class of this bug before it reaches prod.

  4. Audit SCIDEX_STARTUP_REQUIRE_MIGRATIONS=0. The env override exists for emergency boot and will re-enable silent partial- schema serving. Use sparingly.

  5. Cite the M13-4 incident in any PR touching scidex_substrate.core.migrate or _lifespan.

Quick reference — one-liners

# Diagnose
PYTHONPATH=src python3 scripts/ops/check_substrate_health.py --operator-mode

# Dry-run reconcile
PYTHONPATH=src python3 scripts/ops/rescue_substrate_v2.py --dry-run

# Apply reconcile
PYTHONPATH=src python3 scripts/ops/rescue_substrate_v2.py --apply

# Restart + re-check
sudo systemctl restart scidex-substrate-v2
PYTHONPATH=src python3 scripts/ops/check_substrate_health.py