Operations Runbook Index

Operator runbooks for SciDEX-Substrate v2. Each entry covers a specific failure mode, escalation path, and recovery procedure.

Production Readiness

Runbook	Section	Description
Emergency restore	RC §9.6	DB backup restore + federation checkpoint replay
Scheduler down	RC §9.7	No workers running — diagnosis + recovery
Treasury reconciliation drift	RC §9.7	Balance mismatch — reconciliation SQL + correction procedure
Sentinel false-positive	RC §9.7	Quarantine of legitimate content — review + reversibility
Federation peer outage	RC §9.7	Peer sync stalled — peer registry check + re-connect
Isolation-driver leak	RC §9.7	bwrap/namespace leak — detection + cleanup
Migrations	—	Forward-only SQL migration conventions and pitfalls
Monitoring alerts	RC §9.5	Alert index + response procedures for 5 alert surfaces
Watchdogs	SPEC-048	Four operational watchdogs + verb telemetry
Scheduled workers	SPEC-046	Seven LLM-driven content loops — budgets, cadence, audit
Orchestra global pause/drain	SPEC-016	Global fleet pause, timer stop, stale lease reap, and graceful drain verification
Substrate v2 rescue	—	Recovering a broken v2 deployment or migration reconcile failure
Substrate v2 schema reconciliation	—	Schema drift detection + resolution
Federation peer registry	RC §9.2	Published + consumed peer DIDs, registration, conflict resolution

Weekly Ops

Scripts that should be run periodically against the v2 database to maintain data integrity. These are not automated; run manually or add to a scheduled job.

Script	Schedule	Description
`scripts/audit_artifact_links.py`	weekly	Detect dangling `substrate_artifact_links` / `knowledge_edges` refs per SPEC-001 §7.3. `SCIDEX_DSN=postgresql:///scidex_v2 python scripts/audit_artifact_links.py`
`scripts/audit_handle_prefix_convention.py`	weekly	Detect actor handle prefix violations per SPEC-051 §4.1.1.

Domain Operations

Runbook	Section	Description
Persona runtime handoff	—	Queue persona work packets; operate runtimes from scidex-agents
Persona seeding	—	Bootstrap initial personas into the system
Debate orchestrator	—	Four-persona debate panel — run/configure
Debate verdict calibration	SPEC-028 §4	Pantheon resolve verdict accuracy + margin calibration audit
Edge tables migration	—	Schema migration for edge/polymorphic tables

On-call Operations

Document	Section	Description
On-call handover	RC §9.7	Rotation, escalation chain, handover checklist, incident response table, post-mortem process
On-call roster	RC §9.7	Current rotation schedule + specialty contact details
Post-mortems	RC §9.7	Per-incident post-mortem documents + template

Artifact Operations

Document	Section	Description
Paper artifact cache	SPEC-077 §3.7	Paper search/fetch APIs, S3 blob cache contract, v1 cache promotion, live backfill snapshot
Artifact backup readiness	—	Artifact durability audit covering Postgres, local paths, S3, and git-backed bundles
ComputationalReview import session	SPEC-191	Ten-package review import counts, dedupe recommendation refs, source-policy notes, Jerome work packet, and open activation work