Runbook — Monitoring & Alerts

Prometheus/Grafana alert routing and severity ladder.

Source: docs/runbooks/monitoring-alerts.md

Monitoring Alerts Runbook

Section: RC checklist §9.5 Worker: monitoring_alerts (src/scidex_substrate/scheduled_workers/monitoring_alerts/) Escalation: ops_actor (ops_actor system role-agent) On-call: SciDEX ops team

Overview

The monitoring_alerts worker polls 5 alert surfaces every 5 minutes and emits notification events to ops_actor when conditions are met. Alerts are also logged to the scheduled_worker_runs audit trail with fired=True.

Notification channel: ops_actor receives a notification.broadcast event via the SPEC-109 fanout system. The on-call engineer should monitor scidex.notifications.list for recipient_actor=ops_actor or configure a real-time fanout (e.g. PagerDuty webhook, Slack channel) based on the notification kind field.

Alert Index

# Name Trigger Kind Severity
1 Sentinel Quarantine New quarantine_action event in events table warning High
2 Treasury Reconciliation Drift Any row in treasury_reconciliation or budget_reconciliation with drift_usd != 0 warning High
3 Hash-Chain Audit Failure privileged_action_audit chain verification returns tamper signal critical Critical
4 ETL Failure Rate >10% of scidex_etl_runs rows in the last 1 hour have outcome IN ('failed', 'error') warning Medium
5 Integration Smoke Failure Latest integration_smoke_run row has outcome = 'failed' or 'error' critical Critical

Alert Details & Response Procedures

Alert 1 — Sentinel Quarantine

Firing condition: A new row with event_type = 'quarantine_action' has landed in the events table since the last worker run.

Meaning: The Sentinel agent has quarantined one or more artifacts or agents due to detected spam, collusion, or compromised key material.

Response procedure:

  1. Query the quarantine events: SELECT * FROM events WHERE event_type = 'quarantine_action' ORDER BY created_at DESC LIMIT 20;

  2. Identify the affected artifact/agent refs from the event payload.

  3. Follow the Sentinel quarantine review workflow (SPEC-184 §8): list affected artifacts, classify organic vs. suspicious, post a report to Senate.

  4. If key compromise confirmed: follow the key-rotation runbook (SPEC-175 §8).

  5. Acknowledge the alert once the review is underway.

Kind: warning

Alert 2 — Treasury Reconciliation Drift

Firing condition: Any row in treasury_reconciliation or budget_reconciliation has drift_usd != 0.

Meaning: The automated reconciliation job has detected a discrepancy between the recorded budget/quota state and the underlying ledger (artifact signals + compute costs). This could indicate a bug in the reconciliation logic or an accounting discrepancy from concurrent writes.

Response procedure:

  1. Query the drifting row(s):

    SELECT * FROM treasury_reconciliation WHERE drift_usd != 0 ORDER BY computed_at DESC LIMIT 5;
SELECT * FROM budget_reconciliation WHERE drift_usd != 0 ORDER BY computed_at DESC LIMIT 5;

  2. Identify the actor(s) and scope(s) affected.

  3. Check the substrate audit journal for concurrent writes in the affected window: SELECT * FROM substrate_audit_journal WHERE ... (time window of the drift).

  4. If the drift is small (< $1 USD) and the affected actors are test accounts, it may be safe to ignore; document in the ops log.

  5. If the drift is large or growing, escalate to the Treasury lead and pause the affected worker(s).

Kind: warning

Alert 3 — Hash-Chain Audit Failure

Firing condition: The privileged_action_audit hash chain fails verification — a row’s audit_hash does not equal sha256(canonical(row) || prev_audit_hash).

Meaning: A privileged action audit row has been tampered with or a row was inserted out-of-order without updating the chain. This is a critical security signal.

Response procedure:

  1. DO NOT ignore this alert. It indicates potential tampering.

  2. Run a manual chain inspection:

    python -c "
import asyncio
from scidex_substrate.api.privileged_actions import verify_chain, list_audit_log
async def main():
    ok, reason = await verify_chain()
    print(f'chain OK: {ok}, reason: {reason}')
    if not ok:
        log = await list_audit_log(limit=50)
        print(log)
asyncio.run(main())
"

  3. Identify the first inconsistent row (id, executed_at).

  4. Collect the full audit trail from privileged_action_audit for that row’s agent_did and time window.

  5. Contact the security team immediately. Do NOT attempt to repair the chain without security team guidance.

  6. Preserve all logs; do not rotate keys until the incident is resolved.

Kind: critical

Alert 4 — ETL Failure Rate

Firing condition: In the last 1-hour window, more than 10% of scidex_etl_runs rows have outcome IN ('failed', 'error').

Meaning: The ETL ingestion pipeline is experiencing elevated failure. Possible causes: upstream API rate limits, schema changes in v1 source data, network issues, or a bug in a recently deployed mapper.

Response procedure:

  1. List recent failed ETL runs:

    python -c "
import asyncio
from scidex_substrate.skill.verbs.etl_runs_list import etl_runs_list, EtlRunsListIn
from scidex_substrate.skill.types import Context
async def main():
    ctx = Context(...)  # create a system ctx
    result = await etl_runs_list(EtlRunsListIn(outcome='failed', limit_per_table=10), ctx)
    print(result)
asyncio.run(main())
"

    Or via the operator CLI (if authenticated as scidex-admin).

  2. Look for patterns: one specific source table failing, a time window correlating with a recent deployment, or a newly deployed mapper.

  3. Check the errors_sample field on individual rows for the root cause.

  4. If a single source table is failing: pause the affected cron job via its kill switch (SCIDEX_SCHED_<NAME>_ENABLED=false) and file a bug.

  5. If the failure rate is widespread: escalate to the data eng on-call.

Kind: warning

Alert 5 — Integration Smoke Failure

Firing condition: The latest row in integration_smoke_run has outcome = 'failed' or 'error'.

Meaning: The nightly integration smoke test suite has failed. This could indicate a regression introduced by a recent merge or a change in an upstream dependency.

Response procedure:

  1. Query the failing run:

    SELECT * FROM integration_smoke_run ORDER BY started_at DESC LIMIT 1;

  2. Inspect metadata for the pytest output or failure summary.

  3. Identify the failing test(s) from the output.

  4. Check recent merges to main that may have caused the regression.

  5. Post the failure to the #eng-alerts Slack channel with the failing test name and the suspected commit.

  6. If the failure blocks a pending release, coordinate with the release manager before merging further PRs.

Kind: critical

Kill Switch

To mute the worker without stopping the systemd timer:

SCIDEX_SCHED_MONITORING_ALERTS_ENABLED=false python -m scidex_substrate.scheduled_workers monitoring_alerts --dry-run

The worker will log status=disabled but the timer will continue firing, allowing the operator to re-enable without touching cron.

Dry Run

PYTHONPATH=src python -m scidex_substrate.scheduled_workers monitoring_alerts --dry-run

A dry run executes all 5 checks but does not emit notifications or record audit rows. It exits 0 and logs each check result.

Policy Knobs

Env var Default Description
SCIDEX_SCHED_MONITORING_ALERTS_DAILY_USD 0.10 Daily USD cap (bookkeeping only)
SCIDEX_SCHED_MONITORING_ALERTS_PER_RUN_USD 0.10 Per-run USD cap
SCIDEX_SCHED_MONITORING_ALERTS_ETL_THRESHOLD 0.10 ETL failure rate threshold (fraction 0..1)
SCIDEX_SCHED_MONITORING_ALERTS_ETL_WINDOW_HOURS 1 ETL failure rate lookback window
SCIDEX_SCHED_MONITORING_ALERTS_ENABLED (not set) Set to false to disable the worker