On-Call Handover Document

Status: active. Last refreshed 2026-05-16. Owner: Substrate ops + SRE. Reader: Anyone stepping into or out of the on-call rotation for SciDEX-Substrate v2.

Covers RC §9.7 acceptance: on-call rotation, escalation chain, handover checklist, and post-mortem infrastructure. The runbooks (docs/runbooks/) already exist (12 files); this document ties them together with rotation, escalation, and incident-response procedures.

1. On-Call Rotation

Cadence: Weekly, Monday 09:00 UTC to Monday 09:00 UTC.

Roster: Maintained in oncall-roster.md. The current on-call engineer is the first point of contact for all substrate alerts and incidents.

How to reach the on-call engineer:

Slack: #scidex-oncall channel — page with @oncall substrate: <summary>.
Notification verb: scidex.notification.broadcast to ops_actor — the monitoring system already routes alerts here (see docs/runbooks/monitoring-alerts.md).
PagerDuty (TODO): PagerDuty integration not yet wired. Track in SPEC-048 §2.1. Until then, Slack #scidex-oncall is the primary paging channel.

Rotation changes:

Update oncall-roster.md with the new on-call name + dates.
Post a handover message in #scidex-oncall using the checklist in Section 3 below.
Ensure the outgoing on-call has no open P0 incidents (escalate or close before handover).

2. Escalation Chain

Three tiers. Each tier has a target response time. If the current tier does not respond within the window, automatically escalate to the next tier.

Tier	Role	Target response	Reach via
T1	On-call engineer	15 min	Slack `#scidex-oncall`, notification verb
T2	Tech lead / substrate lead	30 min (after T1 escalation)	Slack DM, `@substrate-lead` in `#scidex-oncall`
T3	CTO / VP Engineering	60 min (after T2 escalation)	Phone / direct message; only for P0 with user-facing impact

Specialty escalations

These roles are paged by the on-call engineer when an incident falls within their domain. They supplement the tier chain; they do not replace it.

Domain	Role	Referenced by runbook
Database	DB on-call	`scheduler_down.md` (DB pool exhaustion), `monitoring-alerts.md`
Treasury / finance	Treasury lead	`treasury_reconciliation.md` (drift_usd > $10)
Security	Security team	`monitoring-alerts.md` (hash-chain audit failure)
Data pipeline	Data engineering on-call	`monitoring-alerts.md` (ETL failure rate)
Workers / scheduler	Substrate maintainer	`scheduled_workers.md` (worker error 3+ in 24h)

TODO: Replace placeholder role names with actual handles once the roster is populated. Each row in oncall-roster.md should map the role to a Slack handle and/or PagerDuty contact.

When to escalate

Immediate T2: Any P0 incident (service down, data loss suspected, security breach).
Immediate T3: P0 incident unresolved after 60 min or affecting production data integrity.
Specialty: Any incident that matches the domain table above, regardless of tier.

3. Handover Checklist

Complete every item before transferring on-call responsibility to the next engineer. Post the completed checklist in #scidex-oncall.

#	Item	How to verify
1	No open P0/P1 incidents	Check `#scidex-oncall` for unresolved threads; verify `watchdog_runs` has no `crit` rows younger than 24h
2	Monitor alert history reviewed	Run `scidex.notification.list` for `ops_actor` — review last 7 days for any unacknowledged alerts
3	Treasury reconciliation check	Verify latest `budget_reconciliation` row shows `drift_usd = 0` (or acceptably small). See `treasury_reconciliation.md`
4	Hash-chain verifier last ran within 2h	Check `privileged_action_audit` verification timestamp; if stale, run the verifier manually
5	Slack handover note posted	Summary of the week’s incidents (or “quiet week”), any ongoing watch items, and the incoming on-call’s name
6	Roster updated	Update `oncall-roster.md` with the new rotation dates
7	Scheduler health confirmed	`systemctl status scidex-scheduler` shows active; `scidex_etl_runs` recent rows show `status='ok'`
8	Backup freshness confirmed	Verify the latest PG dump exists and is younger than 25h (see `substrate_v2_rescue.md`)

4. Incident Response Link Table

When an alert fires, use this table to find the correct runbook.

Incident type	Primary runbook	Alert source	Escalation
P0 — substrate down	`substrate_v2_rescue.md`	Watchdog `route_health` crit (502/503/504)	T1 → T2 immediately
Scheduler not running	`scheduler_down.md`	`scidex_etl_runs` stale; systemd unit inactive	T1 → T2 if crash recurs 2+ in 24h
Treasury drift	`treasury_reconciliation.md`	Monitoring alert (treasury reconciliation drift)	T1 → Treasury lead if drift_usd > $10
Hash-chain audit failure	`monitoring-alerts.md` §c	Monitoring alert (hash-chain verification)	T1 → Security team immediately
Sentinel quarantine	`monitoring-alerts.md` §a	Monitoring alert (sentinel quarantine fires)	T1 → review; T2 if false-positive rate > 20%
Worker error loop	`scheduled_workers.md`	`scidex_etl_runs` 3+ errors in 24h	T1 → substrate maintainer
Watchdog failure	`watchdogs.md`	`watchdog_runs` non-ok	T1 → runbook-specific remediation
Federation peer outage	`substrate_v2_rescue.md`	Peer sync stalled	T1 → T2 if >1 peer affected
Isolation-driver leak	`substrate_v2_rescue.md`	bwrap/namespace leak detected	T1 → clean up per runbook

5. Post-Mortem Process

Every P0 or P1 incident that required tier-2+ escalation must produce a post-mortem within 5 business days of incident resolution.

Template

Create docs/operations/post-mortems/YYYY-MM-DD-slug.md using this structure:

# Post-Mortem: <short title>

**Date:** YYYY-MM-DD
**Severity:** P0 / P1
**Duration:** <start time> – <end time> (total: <duration>)
**On-call engineer:** <name>
**Escalation chain activated:** T1 → T2 / T1 → T2 → T3

## Summary

<1-2 paragraph narrative of what happened, from first alert to resolution>

## Timeline

| Time (UTC) | Event |
|---|---|
| HH:MM | First alert fired |
| HH:MM | On-call acknowledged |
| ... | ... |
| HH:MM | Incident resolved |

## Root cause

<What caused the incident. Be specific — cite log lines, metrics, or config values.>

## Action items

| # | Action | Owner | Due | Status |
|---|---|---|---|---|
| 1 | <description> | <name> | YYYY-MM-DD | open |

## Lessons learned

<What went well, what didn't, what we'd do differently>

Process

On-call engineer drafts the post-mortem within 48 hours of resolution.
Tech lead reviews within 72 hours.
Blameless review meeting (optional for P1, required for P0) — schedule within 5 business days.
Action items must have an owner and due date. Track in the post-mortem file and link any resulting Orchestra tasks.

Post-mortem directory

Post-mortems live in post-mortems/. See the README in that directory for indexing conventions.