Runbook — Operational Watchdogs

Detection and remediation flow for substrate watchdog alerts.

Source: docs/runbooks/watchdogs.md

SPEC-048 watchdogs — operator runbook

This runbook covers the four substrate-scoped operational watchdogs and the verb-level tool-invocation telemetry shipped under SPEC-048. None of them are meant to wake an operator at 3 AM by themselves — they write to watchdog_runs (substrate Postgres), and the future Prism alerts dashboard surfaces non-ok rows. This document explains how to read the rows, what each warn/crit means, and how to escalate.

For the spec, see docs/design/spec-048-substrate-operational-watchdogs.md.

Reading the data

Two ways:

-- Recent watchdog runs across all four watchdogs.
SELECT name, started_at, status, action_taken, findings
  FROM watchdog_runs
 ORDER BY started_at DESC
 LIMIT 20;

-- Non-ok runs in the last 6 h.
SELECT name, started_at, status, findings
  FROM watchdog_runs
 WHERE status IN ('warn','crit','error')
   AND started_at > NOW() - INTERVAL '6 hours'
 ORDER BY started_at DESC;

Or through the verb (requires a substrate API session):

curl -s -X POST http://127.0.0.1:8200/api/scidex/watchdog/runs/list \
  -H 'Content-Type: application/json' \
  -d '{"status": "non_ok", "limit": 20}' | jq .

Statuses in watchdog_runs.status:

  • running — the watchdog inserted its row and crashed before terminating it. After ~1 h, treat as error and investigate.

  • ok — nothing of note.

  • warn — degradation, no immediate action.

  • crit — operator should look now.

  • error — the watchdog itself failed (distinct from finding a problem in the system under inspection).

link_checker

  • Cadence: hourly (timer offset HH:07).

  • Service: scidex-watchdog-link-checker.service.

  • What it does: BFS-crawls prism.scidex.ai to depth 3, capped at 200 pages, records HTTP status per URL. No LLM by default (the spec permits a haiku-class root-cause but we shipped without to keep the dependency surface small).

findings shape:

{
  "base_url": "https://prism.scidex.ai",
  "pages_crawled": 47,
  "broken_count": 2,
  "broken": [{"url": "...", "status": 404, "depth": 2, "content_type": "..."}]
}

Severity:

  • ok — zero broken links.

  • warn — 1-3 broken links or any 5xx anywhere.

  • crit — 4+ broken links or any 5xx on the prism index (/).

Escalation:

  • warn: review the broken URLs. A few 404s typically point to a stale Prism alias (compare with SPEC-040’s alias rules) or a deleted artifact still linked from a wiki page.

  • crit with 5xx on /: Prism root is down. Check journalctl -u scidex-prism-v2.service and refer to project_scidex_prism_deploy_race.md (rebuild races are normal for ~30-60 s; sustained 5xx is not).

orphan_checker

  • Cadence: every 30 minutes (timer offsets HH:12 and HH:42).

  • Service: scidex-watchdog-orphan-checker.service.

  • What it does: SQL-only. Three orphan-class probes, each bounded to 1 000 rows so a runaway count never blows up the runner.

findings shape:

{
  "per_class": {
    "analyses_without_render": 0,
    "hypotheses_with_signals_not_on_exchange": 7,
    "kg_edges_not_indexed": 0
  }
}

A null for a probe means it couldn’t run (schema drift / missing table); the runner logs the exception. Treat persistent null for the same probe as a SPEC-048 maintenance task.

Severity:

  • ok — all three probes returned 0.

  • warn — any probe returned 1-19, or any probe is null.

  • crit — any probe returned 20+.

Escalation:

  • analyses_without_render non-zero — usually Prism’s render queue fell behind. Check forge_runs and the analysis-render service.

  • hypotheses_with_signals_not_on_exchange non-zero — filter mismatch in substrate_exchange_index. Run the exchange-rebuild handler.

  • kg_edges_not_indexed non-zero — ETL race. Re-run the affected ETL worker (etl.knowledge_edges).

route_health

  • Cadence: every 8 hours (timer offsets 00, 08, 16:17).

  • Service: scidex-watchdog-route-health.service.

  • What it does: probes 4 representative substrate routes (verb POSTs

    • /health), records status/latency/body-size, compares to the most recent prior ok|warn|crit row.

findings shape:

{
  "n_routes": 4,
  "per_route": [
    {"name": "scidex.audit", "method": "POST", "url": "...",
     "status": 200, "latency_ms": 38, "body_bytes": 1283,
     "severity": "ok"}
  ]
}

Severity (per route, then run = worst per-route):

  • status flip 200 -> 5xx → crit

  • latency this/last >= 2.0 → warn

  • body shrank >= 50 % → warn

  • status non-2xx (no prior) → warn

  • status 5xx (no prior) → crit

Escalation:

  • crit with 502/503/504: substrate API is unhealthy. Check the systemd unit logs, the DB pool (get_db() 5 s tax, see project_orchestra_get_db_5s_tax.md), and recent deploys.

  • warn from latency: re-check on the next run; transient network issues are common at 8 h cadence.

  • Note: this watchdog does NOT kill hung uvicorn (per memory feedback_systemd_restart_always_node_services.md, substrate is on Restart=always). Operators handle restarts.

visual_inspector

  • Cadence: every 6 hours (timer offsets 00, 06, 12, 18:22).

  • Service: scidex-watchdog-visual-inspector.service.

  • What it does: headless chromium screenshots 10 key Prism routes; pixel-diff (Pillow if installed; byte-hash fallback otherwise) vs. baselines committed to data/visual-baselines/.

  • Per memory reference_scidex_bwrap_binary_paths.md: chromium MUST be in /usr/local/bin (copy, not symlink) for the bwrap sandbox.

findings shape:

{
  "n_routes": 10,
  "warn_threshold": 0.05,
  "crit_threshold": 0.20,
  "per_route": [
    {"name": "home", "url": "...", "diff_ratio": 0.013}
  ],
  "output_dir": "/tmp/scidex-visual-diffs"
}

Per route an entry has either:

  • diff_ratio — pixel-delta in [0.0, 1.0].

  • baseline_missing: true — first observation; the candidate is copied to <output_dir>/<name>.candidate.png for the operator to promote.

  • capture_failed: true — chromium absent / timeout.

Severity:

  • ok — all routes < 5 %.

  • warn — any route in [5 %, 20 %), or 1-N missing baselines, or any capture failures.

  • crit — any route >= 20 % diff, or all baselines missing.

Escalation:

  • warn from a single route — eyeball the diff image (saved at <output_dir>/<name>.diff.png). Often a chart re-render that’s benign; promote to baseline if so.

  • crit from home or papers_index — Prism layout regression. Pair with the most recent Prism merge.

  • crit from “all baselines missing” — bad deploy of data/visual-baselines/; restore from git.

Tool-invocation telemetry

Not a watchdog (no systemd timer). The substrate’s verb dispatcher (api/app.py::_make_handler) wraps every spec.func call with the record_invocation middleware from scidex_substrate.watchdogs.telemetry. Each invocation lands one row in verb_invocations (migration 0044) — unless sampling drops it.

Sampling defaults:

  • 100 % of writes (cost_class='write').

  • 10 % of reads (cost_class='read'); override via SCIDEX_TOOL_TELEMETRY_READ_SAMPLE_PCT (0 disables, 100 records all reads).

  • SCIDEX_TOOL_TELEMETRY_DISABLED=1 disables the middleware entirely.

Payload-size cap: 64 KB. If either input or output exceeds the cap the row still lands but input_hash / output_hash are NULL; the metadata.payload_too_large flag is set so consumers don’t conflate “hash absent” with “payload empty.”

Failure modes are deliberately swallowed: a telemetry insert error never propagates to the user. The dispatcher logs the exception via scidex_substrate.telemetry.verb_invocations.

Reading aggregates:

curl -s -X POST http://127.0.0.1:8200/api/scidex/telemetry/verb_invocations/summary \
  -H 'Content-Type: application/json' \
  -d '{"window": "1h", "limit": 50}' | jq .

Response gives per-verb n_calls, n_errors, error_rate, p50_ms, p95_ms, n_reads, n_writes. Empty aggregates is expected on the first hour after deploy, and means either:

  • the verb dispatcher hasn’t seen traffic yet, or

  • verb_invocations migration hasn’t applied (auto-migrate runs on boot; see core.migrate.apply_pending).

Operator install steps

  1. Apply migration 0044_watchdogs_and_verb_invocations.sql (the substrate auto-migrates on boot; this happens automatically on the next deploy cycle, which auto-pulls every 2 minutes per reference_scidex_v2_deploy_pipeline.md).

  2. Install systemd units:

    sudo cp /tmp/scidex-substrate/ops/systemd/scidex-watchdog-*.service /etc/systemd/system/
sudo cp /tmp/scidex-substrate/ops/systemd/scidex-watchdog-*.timer   /etc/systemd/system/
sudo systemctl daemon-reload
for w in link-checker orphan-checker route-health visual-inspector; do
  sudo systemctl enable --now scidex-watchdog-${w}.timer
done

  3. (Optional) seed visual baselines: run the visual_inspector once manually (sudo systemctl start scidex-watchdog-visual-inspector.service), then promote each *.candidate.png from /tmp/scidex-visual-diffs/ into data/visual-baselines/ and commit.

  4. (Optional) tune telemetry sampling for high-load environments:

    sudo systemctl edit scidex-substrate-v2.service
# add: Environment=SCIDEX_TOOL_TELEMETRY_READ_SAMPLE_PCT=2
sudo systemctl restart scidex-substrate-v2.service

Known limitations

  • No alert-table fan-out yet. SPEC-048 §2.1 sketches a richer scidex_alerts table with kind, severity, subject, ack/resolve verbs. We deferred that — a single PR per the prompt; the watchdog_runs table + scidex.watchdog.runs.list verb is the v1 surface.

  • No notifier integrations (Slack/email). Polling scidex.watchdog.runs.list from a Prism dashboard is the v1 path.

  • link_checker doesn’t attempt LLM root-cause.

  • visual_inspector baselines need manual promotion; no UI.