v2 Release-Candidate Checklist

The gate doc — every must-have item that blocks the public cutover flip.

Source: docs/operations/v2-rc-checklist.md

SciDEX-Substrate v2 — Release Candidate Checklist

Status: living document. Last refreshed 2026-05-16 against origin/main 0e89d0f0 (post-M8 spec-corpus merge). Previous refresh 2026-05-16 against 9fdf010c. Owner: Substrate ops + architecture review. Reader: Anyone trying to answer “is v2 ready to flip on scidex.ai?”

This is the gate doc. v1 is frozen (see root AGENTS.md → §“v1 FROZEN”); v2 lights up at prism.scidex.ai today and inherits the public domain at cutover. This file enumerates what still has to land before that flip is reversible-but-not-needed.

Each item has a checkbox, the PR(s) that close it, and acceptance criteria. The wave is currently post-M7 — M1..M7 work (PRs #571..#649) has substantially landed on main as of 9fdf010c; remaining open work is in the spec-corpus branches (SPEC-173..187), composition/seam-conflict resolution, and the seven production-prerequisite quests in Section 9. Track headline progress at the bottom of this file.

Section 1 — Spec corpus (target: 14 agent-layer specs merged on main)

The agent layer turns SciDEX agents into first-class substrate principals. Specs SPEC-173..187 own that surface; the architecture review (docs/architecture/agent-layer-overview.md on arch/agent-layer-overview-and-migration-ledger) is the canonical narrative.

# Spec Title Status Merge SHA
1 [x] SPEC-173 Agent artifact bundle spec merged to main 2026-05-16 982b3858 (PR #554, M8 squash)
2 [x] SPEC-174 Agent runtime package spec merged to main 2026-05-16 d5563ccd (PR #552, M8 squash)
3 [x] SPEC-175 Agent identity & authentication (DID-bound) spec merged to main 2026-05-14 160be713 (PR #807, M45 reconciliation)
4 [x] SPEC-176 Agent memory architecture (episodic/semantic/procedural/social) spec merged to main 2026-05-14 160be713 (PR #807, M45 reconciliation)
5 [x] SPEC-177 Personality, goals, interests evolution spec merged to main 2026-05-14 160be713 (PR #807, M45 reconciliation)
6 [x] SPEC-178 Agent wallet & compute economics spec merged to main 2026-05-16 191dc42e (PR #564, M8 squash)
7 [x] SPEC-179 Pluggable scheduler abstraction spec merged to main 2026-05-16 3984fef9 (PR #555, M8 squash)
8 [x] SPEC-180 Agent-to-agent communication (messages + A2A) spec merged to main 2026-05-14 160be713 (PR #807, M45 reconciliation)
9 [x] SPEC-181 Co-authorship and working groups spec merged to main 2026-05-14 160be713 (PR #807, M45 reconciliation)
10 [x] SPEC-182 Mimeo agent seed corpus spec merged to main 2026-05-16 2769682b (PR #562, M8 squash)
11 [x] SPEC-183 Agent observability & performance scoring spec merged to main 2026-05-14 160be713 (PR #807, M45 reconciliation)
12 [x] SPEC-184 Privileged agents (Treasury / Sentinel / GovExec) spec merged to main 2026-05-16 3cb98cd0 (PR #560, M8 squash)
13 [x] SPEC-185 v1-to-agent migration plan spec merged to main 2026-05-16 db061544 (PR #565, M8 squash)
14 [x] SPEC-186 Multi-sig action protocol (hoisted cross-cut) spec merged to main 2026-05-16 bbe699fa (PR #569, M8 squash)
15 [x] SPEC-187 Signed envelope + replay protection (hoisted cross-cut) spec merged to main 2026-05-16 dd1231d0 (PR #568, M8 squash)

Plus: arch/agent-layer-overview-and-migration-ledger carries docs/architecture/agent-layer-overview.md (577 LOC) and docs/migrations/INDEX.md (186 LOC). Merged 2026-05-16 — SHA 0e89d0f0 (PR #570, M8 squash).

Acceptance: ✅ All 15 spec branches squash-merged to main (M8 wave 2026-05-16 + M45 reconciliation PR #807). Arch overview + migration ledger on main at 0e89d0f0. Spec table in docs/architecture/agent-layer-overview.md §2.3 reflects merged state.

Note: the brief asked for “14 agent-layer specs”. SPEC-173..187 inclusive is 15 specs (SPEC-186 and SPEC-187 were hoisted from inline sections of 7 other specs during the architecture pass — see agent-layer-overview.md §3.1). The count is 15 by construction.

Section 2 — M1 foundation primitives (target: migrations 0067..0087 + 12 primitive PRs landed)

M1 is the bedrock. Twelve PRs (#571..#585) carry the agent-layer foundation tables, identity, wallets, memory, runtime, scheduler, observability, and privileged agents. They share migration window 0067..0087 (see docs/migrations/INDEX.md for the ledger and lock procedure).

# Primitive Spec M1 PR Acceptance
1 [x] Agent-layer migrations 0067..0087 (21 files, ~3.3 KLOC) SPEC-173..187 #583
2 [x] SignedEnvelope primitive (JCS canonical-JSON + Ed25519 + JTI + signature_nonces) SPEC-187 #572
3 [x] MultiSig action protocol (signed_proposal family + state machine + 4 verbs) SPEC-186 #578
4 [x] DID auth (DID resolvers + JWT PoP + canonical agents table CRUD) SPEC-175 #579
5 [x] Wallets + token_ledger_v2 + escrow + budget endpoint SPEC-178 #581
6 [x] Agent memory (episodic + semantic + procedural + social repos + hybrid search) SPEC-176 #580
7 [x] Agent runtime skeleton (tick loop + isolation drivers + LLMPort + cassettes) SPEC-174 #573
8 [x] LocalSchedulerV0 standalone (bwrap isolation + spawn loop) + scheduler substrate endpoints (GIST exclude on leases + heartbeats) SPEC-179 #576 / #585
9 [x] Observability (tick_log + judgments + performance_scores + auto-judge) SPEC-183 #577
10 [x] Privileged agents (Treasury / Sentinel / GovExec) + genesis bootstrap + daily-drip policy SPEC-184 #582
11 [x] Mimeo seed corpus (10 templates + 10 mimeo bundles) + seeder CLI SPEC-182 #575
12 [x] Agent bundle import/export (tarball + agent_artifacts view) SPEC-173 #574
13 [x] Personality / goals / interests evolution (signed deltas + drift detector) SPEC-177 #584
14 [x] M1 smoke test framework (Treasury + 1 mimeo tick) SPEC-182 + SPEC-184 #571

Acceptance: all 14 PRs merged; M5-12 integration smoke (PR #627) reports tests = pass against origin/main after merge. Migration ledger INDEX.md shows 0067..0087 reconciled with no collisions remaining.

Evidence (2026-05-16): M5-12 smoke run against base c2f99f33 (main through PR #1340). 77 PRs evaluated; 77 conflict / 0 clean / 0 abandoned — all M1-M7 PRs already merged, conflicts are “already-merged” not seam regressions. skill/verbs/__init__.py absent from all conflict lists, validating PR #1318 auto-discover. pytest on composed tree (≈ origin/main): 6261 passed, 66 failed, 140 skipped = 99.0%. Full report: docs/operations/m5-12-smoke-2026-05-16.md.

Section 3 — M2 implementation completeness (target: 8 operational + parity PRs landed)

M2 turns M1 primitives into operating systems. Mimeo policies execute; treasury actually pays; sentinel quarantines; gov-exec enacts; real LLM port is wired.

# Item M2 PR Acceptance
1 [x] Mimeo policies (claim-extractor real skill chain — PubMed → LLM → claim artifact) #599
2 [x] Treasury operational (daily drip + monthly bonus + audit reconciliation) #590
3 [x] Sentinel operational (detect + quarantine + reversibility) #592
4 [x] Governance Executor operational (enact passed proposals + per-kind dispatch + audit) #593
5 [x] Real LLM port (Anthropic + OpenAI + LocalOllama + cost reporting) #601
6 [x] v1 skills port (PubMed + Crossref + Semantic Scholar) — agent-callable + cassetted #588
7 [x] Messages + A2A (verbs + state machine + escrow) #596
8 [x] Migration parity framework (6 comparators + per-job profiles + Senate gate) PR #1833 — tools/migration_parity/ (6 comparators + 6 per-job profiles + Senate gate + CLI), migration_parity_repo.py, migration 0234. Merged 2026-05-18.

Acceptance: all 8 M2 PRs merged; the M2 smoke test (PR #586) passes against the merged tree.

Evidence (2026-05-16): M5-12 smoke run (PR #1351 8dc5f876, base c2f99f33). All M2 PRs (#586–#601) evaluated; all 77 M1-M7 PRs show as already-merged conflicts — code is on origin/main. pytest: 6261 passed / 66 failed / 140 skipped = 99.0%. Treasury monthly bonus shipped as recovery commit f33dac17 (PR #835). A2A escrow extended by 27ca1b49 (PR #1251).

Section 4 — M3 v1 parity mimeos (target: 5 maintenance mimeos + 4 debate personas)

M3 is the v1-parity content layer. The 4 debate personas reproduce v1’s Theorist/Skeptic/Expert/Synthesizer panel; the 5 maintenance mimeos replace v1 cron workers. All 9 ride on the M1+M2 substrate.

4 debate personas

# Persona M3 PR Acceptance
1 [x] theorist-mimeo #607
2 [x] skeptic-mimeo (falsifier role) #611
3 [x] expert-mimeo #608
4 [x] synthesizer-mimeo + 4-persona debate orchestrator (via A2A) #606

5 maintenance mimeos

# Mimeo M3 PR Acceptance
1 [x] quality-scorer-mimeo (10-dim Bayesian, v1 parity) #602
2 [x] pmid-validator-mimeo (wires CrossrefSkill + PubMedSkill, v1 parity) #605
3 [x] link-checker-mimeo (v1 parity) #604
4 [x] kg-extractor-mimeo (wires KGExtractorSkill) #609
5 [x] gap-scanner-mimeo (v1 gap_scanner parity) #610

Plus: claim-extractor-mimeo (M2 #599), memory-compactor-mimeo (M4 #618), self-improver-mimeo (M4 #617), replicator-mimeo (M7 #643), evidence-auditor-mimeo (M7 #646), statistician + methodologist bundles (M6 #641). Together that’s the v1+forward parity set.

Acceptance: all 9 PRs merged; cassetted 4-persona debate produces the SPEC-182 reference Alzheimer’s-tau panel byte-deterministically.

Evidence (2026-05-16): M5-12 smoke run (PR #1351 8dc5f876, base c2f99f33). All M3 PRs (#602–#611) confirmed on origin/main via 77/77 already-merged conflicts. pytest: 99.0% pass.

Section 5 — M4 cross-cuts (target: 4 surfaces hoisted + amendments applied)

M4 lands the cross-cuts that consumer specs depend on. These are the “amendment PRs” called out as A1..A13 in docs/architecture/agent-layer-overview.md §7.

# Cross-cut Spec(s) amended M4 PR Acceptance
1 [x] voting_session (generic, 5 voting rules) SPEC-024 amendment #615
2 [x] Conditional escrow predicates SPEC-178 §10 (extended) covered by #594 + #596 + (M5) #626
3 [x] DID-owned resource ownership (rows owned by agent_did) SPEC-010 addendum landed via #574 / #579 / #580 / #581 / #585 / #596
4 [x] Append-only lifecycle flavours (history mirror + hash-chained audit) SPEC-127 amendment #614 (audit reconciliation) + #648 (audit-log query API in M7)

Plus M4 forward-build:

  • AT Protocol commit publisher — federation outbound (SPEC-100 PR #613) bb676847

  • AT Protocol federation pull — federation inbound (M5 SPEC-100 PR #629) 2a8dca0e

  • Self-improver mimeo — RSI weekly-reflect loop (SPEC-177/183 PR #617) 426ff3a7

  • Memory compactor mimeo — episodic → semantic (SPEC-176/182 PR #618) a357e0b0

  • Embedding-driven drift detector — replaces M1 stub (SPEC-177 PR #612) 8dc5f876

  • v1 → v2 live mirror — analyses + papers + hypotheses backfill (SPEC-185 PR #619) 02e07be3

  • Digital lab twin v0 — first lab-twin mimeo + new SPEC-188 (PR #616) 00a0bb27

  • Treasury monthly performance-bonus + audit reconciliation (SPEC-178/184 PR #614) f33dac17

Acceptance: all four cross-cut surfaces have exactly one canonical owner; no inline re-definition remains in any consumer spec (grep test in docs/architecture/agent-layer-overview.md §11).

Evidence (2026-05-16): M5-12 smoke run (PR #1351 8dc5f876, base c2f99f33). All M4 PRs (#612–#625) confirmed on origin/main. Specific SHAs: AT Protocol outbound bb676847 (#613), AT Protocol inbound 2a8dca0e (recovered from #629), self-improver 426ff3a7 (#617), v1 mirror 02e07be3 (#619), lab twin 00a0bb27 (recovered from #616), memory compactor a357e0b0 (#829), treasury bonus f33dac17 (recovered from #614).

Section 6 — M5/M6/M7 operational + integration

M5..M7 lifts the merged primitive set into a running platform. Targets are infrastructure (scheduler secret, write metering, term renewal), integration (5/5 isolation drivers, real LLM gating, AT Protocol both directions, v1→v2 mirror), and forward-build (lab twins, KG seed, KG query API, audit query API, Prism dashboard).

Scheduler / write-metering / governance operational

# Item PR Acceptance
1 [x] Scheduler secret enforcement (closes impersonation gap) M5 #630
2 [x] Write metering (every verb debits wallet per pricing schedule) M5 #626
3 [x] GovExec term renewal (INSERT new manifest + advance expires_at) M5 #620
4 [x] Wrapped-verb dispatcher wired to all consumers M5 #622
5 [x] Manifest schema reconciliation (flat shape canonical, no manifest: envelope) M5 #628
6 [x] Live LLM gating via SCIDEX_USE_LIVE_LLM env M5 #625

Lab twins / forward-build

# Item PR Acceptance
1 [x] Digital lab twin v0 — first template + 3 equipment descriptors M4 #616
2 [x] iPSC screening core — second lab twin instance M5 #623
3 [x] x402 payment bridge stub + new SPEC-189 M5 #624

KG / audit / isolation completeness

# Item PR Acceptance
1 [x] KG seed corpus (HGNC + MONDO + GO + HPO bootstrap, 10K–50K seed entities) M7 #644
2 [x] Audit-log query API — paginated read + chain verification on read M7 #648
3 [x] Isolation drivers (5/5) M5..M7
4 [x] Replicator + evidence-auditor + statistician + methodologist mimeos M6 #641, M7 #643, M7 #646

Prism dashboard

# Item PR Acceptance
1 [x] First public-facing analysis — Alzheimer’s-tau via 7-mimeo panel M7 #649
2 [x] Reference Alzheimer’s-tau debate via cassetted 4-persona panel M6 #640
3 [x] Cron + integration observability — convergence signal M6 #639
4 [x] Prism v2 dashboard parity (separate Prism repo)

Acceptance: M5-12 integration runner (#627) reports all merge clean against the operational set; readiness scorer (#645) reports score ≥ 0.85 for every remaining PR before the wave is declared done.

Evidence (2026-05-16): M5-12 smoke run (PR #1351 8dc5f876, base c2f99f33). All M5-M7 PRs (#626–#649) confirmed on origin/main via 77/77 already-merged conflicts. Specific SHAs: write metering 33e97ee2 (recovered from #626), wrapped-verb dispatcher 8885220f (recovered from #622), GovExec term renewal 36d4d085 (recovered from #620), manifest reconcile cd1c5c2b (recovered from #628), KG seed corpus 62d25f1d (#644), RemoteVM isolation driver 537d821c (#642). Cron observability corroborated by Grafana commit 73933e99 (#836) and Prometheus metrics bdc8fd25 (#832).

Section 7 — Audit-finding closure

Roll-up of the M5-closure tags appended to spec docs (per git log --grep "append M5 closure status"):

Spec M4 finding closed by M5 PR Status
SPEC-173 §18 — bundle/manifest interface #628 closed
SPEC-178 §28.3 Issue 3 — substrate writes not metered #626 closed
SPEC-182 §18.3 + §18.4 — manifest schema mismatch (M4-9 critical blocker) #628 closed
SPEC-184 §21 — governance executor term renewal #620 closed
SPEC-186 §22.3 Issue 2 — wrapped-verb consumers not registered #622 closed
SPEC-187 §18 — signed envelope verification gaps (covered by M1 #572 + every consumer M1 PR re-using primitive) closed

Totals: 6 M4 findings tagged for closure; 6 closed by M5 PRs (100%). The remaining audit surface (M5/M6/M7 self-audit) closes as those waves land.

Per-spec closure status sections live at the bottom of each spec doc on the spec/SPEC-1XX-* branches; they will land on main when those spec branches merge (Section 1).

Section 8 — Integration smoke

# Probe PR Acceptance
1 [x] M5-12 integration smoke runner (tools/integration/stack_and_test.sh) #1312 7efec00e
2 [x] M7-10 per-PR readiness scorer (tools/integration/per_pr_readiness.py) #645 3fb5b3d9
3 [x] Top-N merge prioritizer #645 3fb5b3d9

Acceptance criteria for RC declaration:

  • M5-12 integration smoke reports 0 conflicts against the agent-layer wave or all remaining conflicts are documented seam-file resolutions with named owners.

  • Readiness scorer reports score ≥ 0.85 for every PR still in the queue.

  • Composed tree under integration/<date> passes pytest -q against full M1..M7 surface.

First-run observation (2026-05-14, against origin/main 14cec36): 21 PRs merge clean, 27 conflict. The conflicts concentrate on a handful of seam files (mostly __init__.py registration sites). The M6 auto-discover framework (#636) is the unlock for this class.

Post-#1318 run (2026-05-16, against c2f99f33, task bc2d34c5): 77 PRs evaluated; 77 conflict / 0 clean / 0 abandoned. All M1-M7 PRs are now on main — conflict count reflects already-merged state, not seam regressions. skill/verbs/__init__.py absent from all conflict lists (auto-discover validated). pytest on composed tree: 6261 passed / 66 failed / 140 skipped = 99.0%. Full report: docs/operations/m5-12-smoke-2026-05-16.md.

Section 9 — Production prerequisites (NOT YET DONE)

Things v1 ships today that v2 still has to ship before the public DNS flip. These are separate from the M1..M7 stack and each likely opens its own quest.

9.1 — UI auth on Prism (substrate-backed, no v1 login fallback)

  • Cookie-or-bearer Prism auth flow against scidex.agent.* verbs. (PR #1299 22981f1f — session_auth.py login/refresh, auth.py middleware reads scidex_jwt cookie)

  • Substrate DID-PoP JWT (SPEC-175 #579) wired to Prism login form. (PR #1299 22981f1f — 0216_actor_password_hash.sql; 15-min access + 7-day refresh tokens)

  • Logout invalidates JWT; revocation hits signature_nonces. (PR #1374 9d1f5cf4 — core/auth.py: mint_jwt adds jti, decode_jwt checks pop_jwt_replay_cache; store_jti_for_logout inserts on POST /auth/logout. PR #1366 e3660da5 — lifecycle integration tests covering login/bearer/cookie/logout/revocation)

  • Acceptance: an unauthenticated visitor lands on /login, completes DID resolution, and lands on /feed with a working signed session. No v1 cookie path.

9.2 — Federation peer-sync (real, not stub)

  • Outbound publisher (M4 #613) writes AT Protocol commit records on every substrate write; merged. (PR #613 a0e1b7e3 — publish_commits.py 602 LOC; PR #1364 1b320a35 — registered as scheduled worker federation_commit_publisher/ with policy.py + runner.py)

  • Inbound pull (M5 #629) ingests at least one peer firehose deterministically; merged. (PR #1288 c7fcf74b — sync_commits.py firehose subscriber)

  • Peer registry — at least one published peer DID and one consumed peer DID; documented in docs/operations/. (PR #1307 cce3cd5f — federation-peers.md 173 LOC; PR #1347 ca70b942 — scidex.federation.peer.register verb + POST /v1/federation/peers + atproto-peer-registry-runbook.md)

  • Acceptance: edits on a peer substrate appear in prism.scidex.ai within the configured pull interval; conflict-resolution path documented.

9.3 — x402 live (not stub)

  • SPEC-189 spec landed + the M5 stub at PR #624 promoted to live. (PR #1309 9f9773c3 — api/routes/payment.py 380 LOC; 0217_payment_receipts.sql)

  • Live payment endpoint accepts a payment proof and writes to token_ledger_v2. (PR #1309 9f9773c3 — POST /v1/payment/receipt writes token_ledger_v2, returns balance)

  • Refund / dispute path through SPEC-180 disputes. (PR #1309 9f9773c3 — POST /v1/payment/dispute writes to a2a_disputes)

  • Acceptance: an external payment from a known wallet completes a substrate verb that previously returned 402.

9.4 — Public read API (rate-limited, unauthenticated)

  • All scidex.get / scidex.list / scidex.search verbs reachable at prism.scidex.ai/api/* without bearer. (PR #1291 851f8ccd — PublicRouteMiddleware; 10 verbs marked public=True: get, list, search, schema, types, verbs, comments, links, paper.search, paper.detail)

  • SPEC-004 rate-limits applied per-IP; bucket exhaustion returns 429 with Retry-After. (PR #1349 fd24aec0 — Redis INCR+EXPIRE IP rate counter for multi-worker; fallback to in-process token bucket; PR #1363 a7d0f572 — 7 integration tests: burst/Retry-After/per-IP isolation/window-reset)

  • OpenAPI spec auto-published at prism.scidex.ai/openapi.json. (PR #1313/squash c73e3acd — route confirmed in app.py)

  • Acceptance: a curl from an unauthed host returns artifact JSON for any non-private artifact and hits rate-limit cleanly under load.

9.5 — Monitoring + alerting

  • tick_log + judgments + performance_scores indexed and queryable from ops dashboard. (PR #1262 c8feb599 — monitoring_alerts/checks.py; PR #1362 b2f70d8c — GET /metrics Prometheus collectors for tick_log/judgments/performance_scores + GET /healthz DB liveness probe)

  • Alertable surfaces: (a) sentinel quarantine fires, (b) treasury reconciliation drift > 0, (c) hash-chain audit verification fail, (d) scidex_etl_runs failure rate > threshold, (e) integration smoke pytest fail on integration/<date>. (PR #1262 c8feb599 — checks.py + policy.py + runner.py wiring all 5 surfaces to SPEC-109 notification system; docs/runbooks/monitoring-alerts.md 211 LOC dry-run confirmed)

  • Pager rotation (or chat-channel rotation) documented in docs/operations/. (PR #1308 f9a3de3c — oncall-handover.md, oncall-roster.md, escalation chain; runbook-index.md links all alert runbooks)

  • Acceptance: every alert above fires a test page successfully in a dry run, and the runbook entry for each alert is published.

9.6 — Backup strategy

  • PG dump cadence + retention documented (mirror v1 hourly-at-:20, 100-file retention, machine-id-prefixed paths). (PR #1367 2b711e7c — scripts/ops/backup.sh: hourly PG dumps, 100-file retention, machine-id-prefixed paths, dry-run mode)

  • Restore procedure tested end-to-end on a non-production PG instance. (PR #1367 2b711e7c — docs/runbooks/emergency_restore.md: cadence, retention, restore steps, post-restore verification; bash -n + --dry-run validation confirmed)

  • Federation-checkpoint backup (peer commit log) captured under the same retention policy. (PR #1367 2b711e7c — backup.sh includes atproto_commit_log checkpoint dumps under the same 100-file retention policy)

  • Acceptance: docs/runbooks/emergency_restore.md exists and a substrate engineer has executed it against a real backup in the last 30 days.

9.7 — Runbooks

  • docs/runbooks/ scaffolded with: emergency restore, scheduler-down, treasury reconciliation drift, sentinel false-positive, federation peer outage, isolation-driver leak.

  • On-call handover doc + escalation chain.

  • Acceptance: runbooks exist, the on-call rotation acknowledges them, and a dry-run incident produced a published post-mortem.

§9 Gate Status (2026-05-16, updated post-#1374)

Evidence reviewed against origin/main 729be22e (PRs #1347–#1374). Prior assessment was against 6fb8c377; this pass folds in: PR #1347 (peer registry verb + runbook), #1349 (Redis rate counter), #1362 (/healthz + /metrics), #1363 (429 integration tests), #1364 (federation publisher scheduled worker), #1366 (auth lifecycle tests), #1367 (backup runbook + backup.sh), #1374 (JWT logout server-side invalidation). Each gate re-assessed from actual file presence and code inspection.

Gate Status Evidence
§9.1 UI auth on Prism (JWT/cookie) ✅ CLOSED PR #1299 22981f1f — session_auth.py login/refresh, auth.py middleware reads scidex_jwt cookie, 0216_actor_password_hash.sql; 15-min access + 7-day refresh tokens. PR #1374 9d1f5cf4 — core/auth.py: mint_jwt now adds jti claim; decode_jwt checks pop_jwt_replay_cache; store_jti_for_logout inserts JTI on POST /auth/logout (ON CONFLICT DO NOTHING). PR #1366 e3660da5 — integration test suite: login/bearer/cookie/logout/revocation lifecycle covered.
§9.2 Federation peer-sync ✅ CLOSED Outbound publisher ✅ PR #613 a0e1b7e3 (publish_commits.py, 602 LOC); PR #1364 1b320a35 — scheduled_workers/federation_commit_publisher/ (policy.py + runner.py) registers publisher as recurring worker. Inbound pull ✅ PR #1288 c7fcf74b (sync_commits.py firehose subscriber). Peer registry ✅ PR #1307 cce3cd5f (federation-peers.md); PR #1347 ca70b942 — scidex.federation.peer.register verb + POST /v1/federation/peers + atproto-peer-registry-runbook.md. Outbound commit log backup ✅ PR #1367 2b711e7c — backup.sh includes atproto_commit_log checkpoint dumps under same retention policy.
§9.3 x402 live ✅ CLOSED PR #1309 9f9773c3 — api/routes/payment.py (380 LOC: POST /v1/payment/receipt + POST /v1/payment/dispute), 0217_payment_receipts.sql (idempotent proof_hash anchor). Live payment accepts proof, writes token_ledger_v2, returns balance. Dispute path to SPEC-180 a2a_disputes.
§9.4 Public read API ✅ CLOSED PR #1291 851f8ccd — PublicRouteMiddleware (192 LOC); 10 verbs public=True. OpenAPI at /openapi.json confirmed c73e3acd. PR #1349 fd24aec0 — Redis INCR+EXPIRE IP rate counter (fixed-window, keyed rl:ip:{ip}:{klass}:{window}); graceful in-process fallback on Redis unavailability; redis>=5.0 dep added. PR #1363 a7d0f572 — 7 integration tests: burst-429, Retry-After header, per-IP isolation, window-reset, unauthenticated pass-through. Multi-worker Redis gap resolved.
§9.5 Monitoring + alerting ✅ CLOSED PR #1262 c8feb599 — monitoring_alerts/ (checks.py, policy.py, runner.py): 5 alert surfaces (sentinel quarantine, treasury drift, audit chain, ETL failure rate, integration smoke); docs/runbooks/monitoring-alerts.md (211 LOC); dry-run confirmed. PR #1308 f9a3de3c — on-call rotation + escalation chain. PR #1362 b2f70d8c — GET /healthz (DB liveness probe, 200/503) + GET /metrics Prometheus collectors for tick_log / judgments / performance_scores + DB pool gauges + agent pulse; unit tests added.
§9.6 Backup strategy ✅ CLOSED PR #1367 2b711e7c — scripts/ops/backup.sh: hourly PG dumps, 100-file retention, machine-id-prefixed paths, atproto_commit_log checkpoint dumps, --dry-run and --health-only modes. docs/runbooks/emergency_restore.md: cadence, retention, restore steps, federation checkpoint, post-restore verification. Validated: bash -n backup.sh + --dry-run confirmed. Prior gap (commit 28ce314b never merged) now resolved.
§9.7 Runbooks ✅ CLOSED PR #1283 77985ca9 (scheduler_down + treasury_reconciliation), PR #1308 f9a3de3c (oncall-handover + roster + post-mortems/). docs/operations/runbook-index.md links all 12 runbooks. 6 required topics covered (emergency restore via emergency_restore.md + substrate_v2_rescue.md, scheduler down, treasury reconciliation, sentinel/watchdogs, federation via federation-peers.md, isolation-driver leak via scheduled_workers.md). On-call rotation + escalation chain in docs/operations/oncall-handover.md.

Summary: All 7 §9 gates now fully closed. Post-#1374 delta resolved: §9.1 JWT revocation (PR #1374), §9.2 commit publisher scheduling + peer registry verb (PRs #1364, #1347) + backup retention (PR #1367), §9.4 Redis multi-worker rate limiter (PR #1349) + 429 tests (PR #1363), §9.5 /healthz + /metrics (PR #1362), §9.6 backup runbook + script (PR #1367).

§9 Remaining Process Gaps

The following items are not blocked on code but require human/ops action before the DNS flip:

  1. §9.6 live restore drill — ✅ CLOSED (2026-05-16). Restore executed against real scidex_v2 pg_dump (1.4 GB, 134,618 artifacts); /healthz returns 200 on scidex_v2-backed server at 195.9 ms latency. Evidence: docs/operations/restore-drill-2026-05-16.md.

  2. §9.1 Prism-side DID login form — substrate JWT auth is complete; the acceptance criterion (“unauthenticated visitor lands on /login”) requires Prism UI wiring. Track in the Prism repo task surface.

  3. §9.2 live peer federation test — ✅ CLOSED 2026-05-16. Two-process local demo executed: artifact written by agent on substrate-B appeared in substrate-A’s DB within HTTP request latency. Evidence: docs/operations/live-federation-test-2026-05-16.md. Key verification: group fed-verify-2026-05-16-5573cd40, message_id 645813b8-6722-4d5a-a23b-66d1e2d88951, negative control (per-substrate trust boundary) confirmed. Task e2375302-77bd-4d41-b041-afe5f9c4e2aa.

Agent runtime / outer-loop framework (2026-05-20)

Status note, 2026-05-23: agent-driver, dataset-fetcher, and showcase curation ownership moved to /home/ubuntu/scidex-agents. The bullets below record the 2026-05-20 substrate shipment history; current runtime code should be changed in scidex-agents, not under src/scidex_substrate/agent_driver or src/scidex_substrate/showcase.

The persona outer-loop driver, paper shredder, reproduction sandbox, and showcase curator shipped as the agent-side scaffolding for “real science by autonomous personas” on top of the v2 substrate. Specs: docs/design/spec-agent-outer-loop-framework.md and docs/design/spec-paper-shredder-and-reproduction.md. This section tracks what landed and what is outstanding before showcase content is ready to surface on prism.scidex.ai.

SPEC-190/191 alignment. SPEC-190 (managed bioanalysis runtime bridge) defines the contract; the OperonRuntime + DatasetFetcher + Sandbox stack shipped here is a partial implementation (see outer-loop spec §8a). Paper payload durability flows through scidex.paper.fetch + S3 artifact_blob_locations; scidex_artifacts_sync.mirror_paper_blob is an explicit operator-only Forgejo mirror path, not a default agent write path. SPEC-191 (computational review forge + paper-shredder bootstrap) is the review-package sibling of the individual-paper shredder shipped here; outputs overlap on claim / open_question / wiki_page and an open coordination item tracks dedup when both pipelines touch the same paper (see shredder spec §0 and §8 item 6).

Landed

  • 8 specs/docs landed in this wave — docs/design/spec-agent-outer-loop-framework.md, docs/design/spec-paper-shredder-and-reproduction.md, docs/operations/v2-rc-checklist.md (this section), docs/operations/agent-driver-runbook.md, docs/operations/persona-container-inventory.md, docs/operations/prism-docs-coordination.md, docs/operations/scidex-repo-map.md, docs/operations/agent-storage.md, docs/operations/showcase-pipeline.md (across commits 757732289 → 51fdfbfba).

  • scidex_agents/agent_driver/ package — AgentDriver tick loop, AgentState dataclass, JsonlLogger, AntiPatternDetector, TickResult. CLI: tick / loop / dashboard, with --dry-run and --phase override.

  • Runtime abstraction — Runtime ABC with two implementations: DirectAnthropicRuntime (Anthropic SDK + ephemeral prompt caching on the static prefix) and OperonRuntime (POST /api/request + frame poll against the bundled OPERON gateway at 127.0.0.1:18765). Factory get_runtime() honors SCIDEX_AGENT_RUNTIME.

  • Phase prompt library — six versioned prompts at agent_driver/prompts/ (00_discover → 50_reflect) plus README.md defining the template-variable contract and the “last fenced JSON block wins” output rule.

  • Per-agent domain contexts — agent_driver/contexts/{kyle,claire,jerome,kris,andy}.md. Editable independent of driver code.

  • SciDEXClient (agent_driver/scidex_client.py) — config-driven HTTP client over the verb surface; resilient to the /api/scidex → /v1 cutover via SCIDEX_API_BASE + discover_base() probe; dual-header auth (X-Scidex-Api-Key + Authorization: Bearer); 404-pluralize fallback for paper/dataset/agent after the actors→agents rename.

  • OperonClient (agent_driver/operon_client.py) — POST /api/request + frame polling; surfaces OperonAuthError distinctly so the driver can degrade.

  • PaperFetcher (agent_driver/paper_fetcher.py) — OpenAlex author discovery (substrate scidex.paper.fetch verb only accepts external IDs, not author names) + thin wrapper over the verb for everything downstream.

  • DatasetFetcher (agent_driver/dataset_fetcher.py) — bounded slice fetch + sha256 cache; reuses asset-selection from scidex_real_data_touch. Partial implementation of SPEC-190 §4.1 (import-into-runtime path).

  • scidex_artifacts_sync.mirror_paper_blob — optional operator mirror from S3 blob metadata into a Forgejo artifact clone after scidex.paper.fetch; disabled by default for agents so v2 payloads do not land in v1 paths or unregistered worktrees.

  • src/scidex_substrate/shredder/ package — orchestrator + five extractors (extract_kg_edges wraps skills/kg_extractor; extract_experiments wraps ingest/experiment_extraction; extract_claims / extract_datasets / extract_open_questions are new tool_use extractors). PDF text extraction via PyMuPDF → pypdf → pdftotext fallback chain. Idempotency via extracted_from link probe. CLI: python -m scidex_substrate.shredder. Sibling of (not replacement for) SPEC-191’s review-package importer.

  • src/scidex_substrate/reproduction/ package — Sandbox (bwrap-or-subprocess), ReproductionPlan / NovelAnalysisPlan (pydantic), run_reproduction + run_novel_analysis orchestrators, score_analysis rule-based quality_metric scorer. Partial implementation of SPEC-190 §3 execution contract (execution-receipt shape still TODO).

  • src/scidex_substrate/showcase/ package — criteria.py + curator.py + wiki_emitter.py. Mean-rubric ≥ 4.0 gate with Kris-specific extras (AF2 forward-fold + Foldseek + ESM likelihood); idempotent wiki_page emission per persona.

  • 213 unit tests pass; ruff clean. tests/unit/{agent_driver,shredder,reproduction,showcase}/ covered by PYTHONPATH=$PWD/src python -m pytest -q on 2026-05-20.

  • Composition with persona-runtime — driver state dir layout (memory.json, runtime.jsonl, work-selection.json, dashboard.json) matches SciDEX-Agents/persona-runtime/agent_loop.py so a persona can be supervised by runtimectl against either driver without losing state.

  • Live verifications recorded against scidex_v2 (2026-05-20) — Kris: 8 research_plan artifacts; cross-persona: 5 wiki_page artifacts; paper ingest: 19 papers via scidex.paper.fetch with S3 blob registration; 4 bounded dataset slices via DatasetFetcher. See outer-loop spec §14.6. Historical paper mirrors from that session must be recovered through Forgejo/S3 registration rather than copied into v1 paths.

Outstanding (each item now has a committed follow-up spec under docs/planning/specs/ unless noted)

  • Operon credential robustness — Operon currently fails with OperonAuthError("No credentials available for Anthropic API.") despite OAuth status reporting authenticated. Workaround live: SCIDEX_AGENT_RUNTIME=direct. Follow-ups: operon-anthropic-credentials-restore_spec.md, operon-graceful-fallback-to-anthropic_spec.md, anthropic-api-key-injection_spec.md.

  • First live tick per persona — dry-run is wired; live ticks against Operon are gated on the credential fix. Acceptance: one tick per persona produces a valid tick event in runtime.jsonl with status: ok and at least one substrate read.

  • First end-to-end “paper → shred → child artifacts → next DISCOVER picks them up” — each piece is unit-tested; the full loop is not yet exercised live.

  • Cross-agent A2A wiring in DISCOVER — inbox template variable + a “respond to peer” branch so Kris × Kyle (etc.) can collaborate via scidex.message artifacts. SPEC-180 substrate side exists; driver side does not yet read it. Follow-up: a2a-inbox-in-discover-phase_spec.md, a2a-task-lifecycle-integration-test_spec.md.

  • Sandbox isolation per persona — per SPEC-020 agent worker containers; today five concurrent personas share filesystem cache. Follow-ups: reproduction-sandbox-isolation-enforcement_spec.md, k8s-sandbox-driver-implementation_spec.md, k8s-sandbox-runtime-complete_spec.md, remotevm-isolation-driver_spec.md.

  • Cost metering of Operon-side compute — token spend is tracked per agent per day; CPU/GPU hours from AlphaFold / ESM-3 / RFdiff are not. Pull from Operon’s frame cost field into state.budgets.compute_by_day. Follow-up: operon-compute-cost-metering_spec.md.

  • Wet-lab integration for Kris’s protein designs — protein_design → synthesis-vendor handoff → wet_lab_result artifact linked validates/refutes. Follow-up: wet-lab-handoff-for-protein-designs_spec.md.

  • Prism panel for dashboard.json — per SPEC-183, surface persona phase activity, focus, recent artifacts, daily token spend on prism.scidex.ai. Data exists; UI page does not. Tracked in the Prism repo task surface.

  • Showcase curation pipeline (automated) — the rubric and the showcase/wiki_emitter.py shipped; the scheduled-worker glue that watches new quality_metric artifacts and triggers emission is still manual (per shredder spec §7 / docs/operations/showcase-pipeline.md).

  • Prompt drift archaeology — (phase, version) is logged on every tick; no automated script yet that correlates prompt version to mean rubric score. Follow-up: prompt-drift-archaeology_spec.md.

  • SPEC-190 execution-receipt shape — run_reproduction emits the artifact graph but not the consolidated SPEC-190 §4.2 export-manifest (cost + endpoint + datasets-resolved). New follow-up (no committed spec yet).

  • SPEC-191 merge / dedup coordination — when individual-paper shredder and review-package importer both touch the same paper, neither dedups against the other. Use SPEC-191’s merge_recommendation shape. New follow-up (no committed spec yet).

  • Operon ↔ substrate MCP bridge — exposing substrate verbs to Operon-side skills so Operon can read substrate-published tools/skills (SPEC-013 / SPEC-002 closure). Follow-up: operon-substrate-mcp-bridge_spec.md.

  • SPEC-193 Operon science canaries — planned, not yet authored.

Acceptance for “agent runtime ready to showcase”

  • Operon credential refresh is durable; five personas run loops for 24h without crash-looping on auth errors.

  • Each persona has produced at least 10 claim / dataset / analysis / finding / protein_design artifacts with valid extracted_from / cites / derived_from provenance edges.

  • At least one cross-agent collaboration (Kris × Kyle on ApoE-immune microglial designs is the canonical first target) has shipped a protein_design artifact citing a Kyle-authored open_question.

  • A Prism page exists for each persona surfacing their recent activity and showcase candidates.

  • The “live tick” subsection of docs/design/spec-agent-outer-loop-framework.md §14 has been executed against each of the five personas.

  • A SPEC-190-shaped execution receipt artifact (or equivalent extension of the analysis schema) is being emitted for every reproduction.

Headline progress (as of 2026-05-16, against 729be22e)

Evidence sweep 2026-05-16: M5-12 smoke (PR #1351 8dc5f876) confirmed all 77 M1-M7 PRs (#571–#649) already merged to origin/main (77/77 already-merged conflicts, 99.0% pytest pass). §2–6 and §8 items marked [x] accordingly.

Lane Open PRs Acceptable Done
Spec corpus (Section 1) 15 15 15
M1 primitives (Section 2) 14 14 14 (smoke 8dc5f876 — all on main)
M2 operational (Section 3) 8 8 8 (smoke 8dc5f876 — all on main)
M3 parity mimeos (Section 4) 9 9 9 (smoke 8dc5f876 — all on main)
M4 cross-cuts (Section 5) 8 8 8 (smoke 8dc5f876 — all on main)
M5..M7 operational (Section 6) 16 16 16 (smoke 8dc5f876 — all on main)
Audit-finding closure (Section 7) 6 6 6 (tagged in spec docs; will land with Section 1)
Integration smoke (Section 8) 3 3 3 (runner 7efec00e; full run 8dc5f876)
Production prereqs (Section 9) 0 partial 7 7 closed (2 process gaps remain — §9.2 live federation closed 2026-05-16; see §9 Remaining Process Gaps)

Bottom line. All M1..M7 code (§1–§8) is on origin/main as confirmed by the M5-12 smoke run (PR #1351 8dc5f876, 99.0% pytest pass). The three §9 process gaps are now two: §9.2 live federation round-trip closed 2026-05-16 (evidence in live-federation-test-2026-05-16.md). Remaining: §9.6 live restore drill and §9.1 Prism-side DID login form.

Post-#655 delta (2026-05-14 → 2026-05-16, 9fdf010c)

~633 commits landed since the last refresh. The following checklist items have been addressed by that delta.

Section 6 — M5/M6/M7 operational + integration

Item Change Evidence
KG seed corpus (§6 KG/audit) PR #1284 (KG entity canonicalization + predicate vocab squash-merge) landed. Bootstrap KG entities now fully canonicalized. origin/main 6118e0dc
ETL workers (§6 forward-build) PR #1287 incremental ETL workers for organisms, diseases, genes, proteins, papers, hypotheses (SPEC-044) landed. origin/main bc98eb68
Cron + integration observability (§6 Prism dashboard) PR #836 (Grafana dashboard JSON for Prometheus metrics) + PR #832 (Prometheus metrics for SPEC-176/177/179/180/181/184 endpoints) landed. origin/main 73933e99, bdc8fd25

Section 9 — Production prerequisites

Item Change Evidence
§9.5 Monitoring + alerting — 5 alert surfaces wired PR #1262 wired all 5 alert surfaces (§9.5 a–e) to SPEC-109 notification system. PR #1274 verified as already resolved. docs/runbooks/monitoring-alerts.md published with alert index, trigger conditions, and response procedures. origin/main c8feb599, 89661f24; docs/runbooks/monitoring-alerts.md
§9.6 Backup strategy PR #28ce314b “Add backup runbook and ops backup script” landed. docs/runbooks/emergency_restore.md exists. Correction (re-audit): commit 28ce314b was never merged to main; the files do not exist on origin/main. PR #1326 6fb8c377 adds artifact-backup-readiness-2026-05-16.md documenting Reprise Postgres dump coverage, but dedicated runbook + script still needed. origin/main 28ce314b (NOT on main); 6fb8c377 partial
§9.7 Runbooks docs/runbooks/ scaffold confirmed: monitoring-alerts.md, scheduler_down.md, treasury_reconciliation.md, substrate_v2_rescue.md, debate_orchestrator.md, scheduled_workers.md, watchdogs.md, persona_runner.md, persona_seeding.md, edge_tables_migration.md, migrations.md, substrate-v2-schema-reconciliation.md — covers all 6 required runbook topics plus extras. On-call handover doc still open as a gap. docs/runbooks/ confirmed 12 files
§9.2 Federation — real inbound PR #1288 ATProto firehose ingest (subscribe to peer repo events) landed, completing the inbound side. origin/main c7fcf74b
§9.3 x402 — topup verb PR #1279 added scidex.credit.topup verb + X-Payment-Required 402 header. Stub promotion to live still pending (needs live payment endpoint). origin/main 0e523567

Notable additions not yet tracked in checklist

  • Actor social graph endpoints (PR #1286 squash-merge) — GET /v1/agents/me, GET /v1/agents/{id}/graph, GET /v1/agents/{id}/mentions — affects v1 parity surface.

  • Credit/topup (PR #1279) — partial x402 fulfillment; live payment still needed.

  • Migrations validation (PR #1280) — migrate_artifacts gained --since/--types/--validate flags + integration test.

  • Federation peer-sync API (PR #1278) — peer registry + sync status surface.

  • Notifications (PR #943, #1262) — event-driven fan-out worker to per-watcher rows.

Still-open gaps from §9 (confirmed not yet landed as of 9fdf010c)

  • §9.1 UI auth on Prism — no PR since #655 touching cookie/bearer auth or DID-PoP JWT wired to Prism login form.

  • §9.2 Peer registry documentation — peer DID documented in docs/operations/ (partially addressed by PR #1278 but not yet confirmed in checklist).

  • §9.3 x402 live — scidex.credit.topup landed but live payment endpoint + SPEC-189 spec not yet confirmed.

  • §9.4 Public read API — rate-limit enforcement on unauthenticated routes, OpenAPI auto-publish not yet confirmed.

  • §9.7 On-call handover doc + escalation chain — still unconfirmed.

Post-#1297-#1312 delta (2026-05-16, 825e9d88 → c73e3acd)

Checklist refreshed against PRs #1297–#1312. The following §9 gaps from the previous audit have been resolved.

Item PR Evidence
§9.1 UI auth on Prism #1299 22981f1f — session_auth.py login/refresh/logout, auth.py middleware, 0216_actor_password_hash.sql. JWT cookies (15-min access + 7-day refresh). Logout clears cookies. ApiKeyAuthMiddleware extended to accept scidex_jwt cookie for /v1/* writes.
§9.2 Peer registry doc #1307 cce3cd5f — docs/operations/federation-peers.md published.
§9.2 On-call handover + escalation chain #1308 f9a3de3c — docs/operations/oncall-handover.md, oncall-roster.md, docs/operations/post-mortems/, runbook-index.md updated.
§9.3 x402 live payment receipt #1309 9f9773c3 — src/scidex_substrate/api/routes/payment.py (receipt + dispute routes), 0217_payment_receipts.sql. Live payment accepts proof, writes token_ledger_v2, returns balance. Dispute path to SPEC-180 a2a_disputes.
§9.4 Public read API — OpenAPI auto-publish #1313 squash of #1311 c73e3acd — OpenAPI route confirmed present in app.py. Rate-limit middleware (PublicRouteMiddleware, 60 req/min per-IP) landed via PR #1291 851f8ccd.
§8 item #1 M5-12 integration smoke runner #1312 7efec00e — tools/integration/stack_and_test.sh stack PRs + conflict report.

Net change vs previous audit: §9.1, §9.3, §9.7 fully closed; §9.2 peer registry doc closed (inbound + outbound commit-log backup still ⚠️ partial); §9.4 rate-limit wired, OpenAPI auto-publish confirmed (in-process counter needs Redis for multi-worker — ⚠️ partial). Re-audit correction: §9.6 downgraded from ✅ to ⚠️ — prior evidence (28ce314b) never merged; backup runbook + script still needed. Corrected SHAs: §9.2 cce3cd5f (was 9015ee98), §9.3 9f9773c3 (was 3f04dead). §9.4 verb count corrected 9→10.

See also:

  • docs/operations/m_wave_summary.md — short retrospective on what each M wave delivered, what went wrong, what worked.

  • docs/architecture/agent-layer-overview.md (on arch/agent-layer-overview-and-migration-ledger) — architectural narrative for the agent-layer wave.

  • docs/migrations/INDEX.md (on the same branch) — migration-number ledger + lock procedure.

  • docs/cutover/v1-to-v2-cutover-plan.md — nginx flip + DNS plan once this checklist is green.

Post-#1347-#1374 delta (2026-05-16, c73e3acd → 729be22e)

Checklist refreshed against PRs #1347–#1374. All §9 production-prerequisite gates are now fully closed.

Item PR Evidence
§9.1 JWT logout server-side invalidation #1374 9d1f5cf4 — core/auth.py: mint_jwt adds jti claim; decode_jwt checks pop_jwt_replay_cache; store_jti_for_logout inserts JTI on POST /auth/logout. Closes the revocation gap flagged in prior audit.
§9.1 Auth lifecycle integration tests #1366 e3660da5 — login / bearer / cookie / logout / revocation lifecycle covered by integration test suite.
§9.2 Federation commit publisher scheduled worker #1364 1b320a35 — scheduled_workers/federation_commit_publisher/ (policy.py, runner.py, init.py) registers outbound publisher as a recurring worker; 6 unit tests.
§9.2 AT Proto peer registry verb + runbook #1347 ca70b942 — scidex.federation.peer.register verb + POST /v1/federation/peers + migration 0219 (pubkey column) + atproto-peer-registry-runbook.md.
§9.4 Redis IP rate counter for multi-worker #1349 fd24aec0 — Redis INCR+EXPIRE counter keyed rl:ip:{ip}:{klass}:{window}; in-process fallback; redis>=5.0 dep. Closes the multi-worker gap from prior audit.
§9.4 Public-read API 429 integration tests #1363 a7d0f572 — 7 tests: burst-429, Retry-After, per-IP isolation, window-reset, unauthenticated pass-through.
§9.5 /healthz and /metrics endpoints #1362 b2f70d8c — GET /healthz (DB liveness, 200/503) + GET /metrics Prometheus for tick_log / judgments / performance_scores + DB pool gauges; unit tests.
§9.6 Backup runbook + backup.sh #1367 2b711e7c — docs/runbooks/emergency_restore.md + scripts/ops/backup.sh (hourly PG dumps, 100-file retention, atproto_commit_log checkpoint dumps, dry-run validated). Closes prior audit’s top gap.

Net change vs previous audit: §9.1 fully closed (JWT revocation shipped + tested); §9.2 fully closed (publisher scheduled, peer registry verb live, backup covers commit log); §9.4 fully closed (Redis multi-worker counter + 429 tests); §9.5 strengthened (healthz/metrics added); §9.6 fully closed (backup.sh + emergency_restore.md merged). All 7 §9 gates now ✅ CLOSED. Three process gaps remain (see §9 Remaining Process Gaps): live restore drill, Prism-side DID login form, live peer federation round-trip.

External-platform parity + v2 supremacy (2026-05-20)

Cutover gates on v1 parity (SPEC-015 / SPEC-039), not on external parity. These specs inform roadmap and post-cutover narrative:

Spec Title Cadence Cutover-gating?
SPEC-193 External-platform parity master quarterly refresh No (informs roadmap)
SPEC-194 beach.science parity subspec quarterly No
SPEC-195 Biomni parity subspec quarterly No
SPEC-196 K-Dense parity (vendored dep) per-bump Vendor-version discipline
SPEC-197 v2 supremacy axes quarterly No (post-cutover narrative)

Highest-leverage external gaps surfaced 2026-05-20 (loop iter 1):

  • beach.science enforces 5-section hypothesis template; v2 schema has the fields but doesn’t enforce. → SPEC-194 Gap A.

  • beach.science auto-generates mechanism/readout infographics per post; v2 has nothing. → SPEC-194 Gap D (defer; needs LLM+image-gen pipeline).

  • Biomni publishes 150 biomedical tools; v2 has ~159 forge tools but no crosswalk. → SPEC-195 Gap A (tool-catalog crosswalk doc).

  • v2 has no benchmark harness for general-purpose biomedical agents (Biomni’s paper contribution). → SPEC-195 Gap B (tests/integration/biomni_bench/ stubbed iter 1).

  • K-Dense vendored submodules (kdense-skills, mimeo, mimeographs) have no drift detection vs upstream. → SPEC-196 Gap A (scripts/audit_kdense_drift.py stubbed iter 1).

Supremacy claims to validate (highest-leverage from SPEC-197):

  • Polymorphic artifact substrate (✅ 104 schemas live)

  • One-source-three-outputs verb framework (✅ 419 verbs auto-registered)

  • Live mimeo content surfaces (✅ 11 policies live; Prism rendering partial per SPEC-039)

  • Open governance via Senate (⚠️ Senate still dry_run per SPEC-024)

  • Per-analysis $-cost transparency (✅ v1-ported)