SPEC-008 — Observability

Metrics, traces, structured logs, and the v2 event stream.

Source: docs/design/spec-008-observability.md

SPEC-008 — Observability

Field Value
Status Draft v1
Owner kris.ganjam@gmail.com
Date 2026-04-28
Depends on SPEC-001, SPEC-004
Pillar Senate

TL;DR

Three pillars: structured logs (JSON, request_id correlation), metrics (Prometheus, per-verb counters and latencies, bucket levels), traces (OpenTelemetry, request → service → DB span). Plus the existing db_write_journal as an audit log of record. Dashboards live in Grafana with subscribe-driven panels embedded in Prism. Alerts route to a single rules file; existing fleet-health watchdog stays as the bottom-line liveness check.

1. What we have today

  • Structured-ish logs (mixed text and dict).

  • db_write_journal table — full audit of every mutation (per-row before/after JSON, agent_id, task_id, model, slot_id).

  • events table — pub/sub backbone, also doubles as event-log archive.

  • Fleet health watchdog (Claude-powered self-healing, systemd timer every 5m). See memory entry reference_fleet_health_watchdog.md.

  • No formal metrics or traces.

2. Why now

The polymorphic substrate adds new failure modes:

  • A misbehaving validator could reject a category of writes silently.

  • A new bucket scope could throttle agents unexpectedly.

  • Embedding worker drift (semantic search) could silently degrade mode=auto results.

  • A subscribe consumer disconnect could cause event backpressure.

Without observability, all of these are detected only by user complaint. The substrate is too important and the surface too broad.

3. Pillar 1 — Structured logs

3.1 Format

Every log line is a JSON object:

{
  "ts":         "2026-04-28T19:42:11.234Z",
  "level":      "info",
  "request_id": "req-uuid",
  "agent_id":   "agent-uuid",
  "verb":       "scidex.create",
  "type":       "hypothesis",
  "duration_ms": 87,
  "outcome":    "success",
  "msg":        "create succeeded",
  "extra":      { /* verb-specific */ }
}

3.2 Correlation

request_id is generated at HTTP/MCP boundary and propagated through every log line, error, event payload, and journal entry. A single failure in production is reconstructable by grep request_id.

3.3 Levels

  • error — request failed unexpectedly. Always paged.

  • warn — degraded behavior (validator shadow rejection, fallback activated).

  • info — verb invocations, lifecycle transitions.

  • debug — verbose tracing (off by default; per-agent flag to enable).

3.4 Storage

Local rotating files; nightly ship to a long-term store (existing infra). Retention 30 days hot, 1 year cold.

4. Pillar 2 — Metrics

4.1 Counters

Counter Labels
scidex_verb_calls_total verb, type, outcome
scidex_validation_failures_total verb, type, validator, path_prefix
scidex_version_conflicts_total type, mergeable
scidex_rate_limit_rejections_total scope, kind
scidex_events_published_total event_type
scidex_events_delivered_total event_type
scidex_subscribe_active_consumers (gauge)
scidex_artifacts_count type (gauge, sampled hourly)

4.2 Histograms

Histogram Labels Buckets
scidex_verb_duration_seconds verb, type 5ms..10s
scidex_validation_duration_seconds validator 0.1ms..1s
scidex_db_query_duration_seconds query_kind 1ms..5s
scidex_event_publish_to_deliver_seconds event_type 1ms..1min

4.3 Gauges

Gauge Labels
scidex_rate_limit_bucket_tokens agent_id, scope
scidex_quota_remaining agent_id, scope, period
scidex_budget_remaining_usd agent_id, scope, period
scidex_subscribe_lag_events consumer_id
scidex_embed_queue_depth

4.4 Endpoint

GET /metrics exposes Prometheus text format. Auth: substrate-internal token. Scraped by Prometheus every 30s.

5. Pillar 3 — Traces

5.1 Spans

Each verb invocation creates a root span with attributes:

verb=scidex.create
type=hypothesis
agent.id=...
agent.kind=ai_local
session.id=...
delegation_chain=[...]
request_id=...

Child spans:

  • validate.<validator_name> (per validator)

  • db.write.artifacts, db.write.history, db.write.signals

  • event.publish.<event_type>

  • forge.dispatch.<tool_name> (when applicable)

5.2 Backend

OpenTelemetry SDK exports OTLP to a local collector → ship to Jaeger or similar (existing infra). Sampling: head-based at 10% by default; 100% for error and version_conflict.

5.3 Why traces matter for an agent-first substrate

A common debug question: “Why did this agent’s task take 30s?” Without traces, you grep logs and reconstruct a sequence. With traces, you click one trace ID and see the full call tree, including the embedding worker call, the validator that ran, the DB row-lock wait. For agent-driven workflows where one task spawns many tool calls, this matters.

6. The audit log of record: db_write_journal

Already exists. Every write captures:

  • entity_key, before_json, after_json, changed_fields

  • agent_id, slot_id, model, task_id, commit_hash

  • reason, source, created_at

This is not a metrics or log surface — it’s a forensic trail. It survives log rotation. Audits, dispute resolution, and post-incident reconstructions read from this.

The polymorphic verb layer extends db_write_journal with:

  • delegation_chain (full, JSONB)

  • idempotency_key

  • request_id (for cross-correlation)

This is the single source of truth for “what happened, who did it, when, and why.”

7. Dashboards

7.1 Grafana panels

A starter dashboard:

  • Overview — total verb calls/min, error rate, p99 latency.

  • By verb — call rate and error rate per verb.

  • By type — write rate per artifact type.

  • Validation — rejections per validator. Spike means bad agent or bad validator.

  • Locking — version_conflict rate. Spike means contention.

  • Subscribe health — active consumers, lag distribution.

  • Embed worker — queue depth, embed rate.

  • Budgets — top 10 agents by spend in the current period.

7.2 Embedded panels in Prism

Senate Quality Dashboard is in Prism, not Grafana. It subscribes to events:

  • agent.budget_exceeded — alert tile.

  • system.budget_drift_detected — alert tile.

  • Rolling rate of validation_failed per validator — line chart.

  • Recently superseded artifacts with reasons — feed.

The same data drives both. Grafana for ops; Prism for governance/visibility.

8. Alerts

A single alert rules file (monitoring/alerts.yaml):

Alert Condition Severity
SubstrateDown up{job="scidex"} == 0 for 2m page
Verb5xxBurst rate(scidex_verb_calls_total{outcome="internal_error"}[5m]) > 0.1 page
LatencyP99Regress histogram_quantile(0.99, scidex_verb_duration_seconds) > 1 for 10m warn
ValidationStorm rate(scidex_validation_failures_total[5m]) > 5/s warn
ConflictStorm rate(scidex_version_conflicts_total[5m]) > 1/s warn
EmbedQueueBacklog scidex_embed_queue_depth > 1000 for 30m warn
BudgetDrift system.budget_drift_detected event warn
SubscribeConsumerStuck scidex_subscribe_lag_events > 10000 for 10m warn
ActorRunaway single agent exceeds 90% of any budget in <1h page (alert name retained from v1; covers any agent kind)

Routes: page → on-call; warn → log + Senate dashboard tile.

9. Health checks

9.1 /health

  • up — process alive.

  • db_reachable — can SELECT 1.

  • events_writable — can INSERT to events.

  • embed_worker_alive — last heartbeat within 60s.

  • forge_reachable (if Forge is configured).

Returns 200 when all green; 503 with structured body listing failures otherwise.

9.2 Fleet watchdog integration

Existing orchestra-fleet-watchdog.service polls /health every 5 minutes. On red, it triggers a Claude-driven self-heal flow (see memory reference_fleet_health_watchdog.md). No change needed.

10. Privacy

Logs and metrics MUST NOT carry:

  • Passwords, JWTs, API keys, OAuth tokens.

  • Raw user input from comments/wiki content beyond a 200-char snippet.

  • Specific patient data or identifiable PII (the corpus is mostly public papers, but some artifacts are author-private — gate them).

A scrubber middleware (scidex.observability.scrub) runs on every log emission and metric label.

11. Migration plan

# Title Scope Risk
O1 Structured-log middleware + request_id propagation Replace ad-hoc logging in api.py and service layer. Low
O2 Prometheus /metrics endpoint + initial counters/histograms New endpoint; no behavior change. Low
O3 Bucket/quota/budget gauges Hooks into rate-limit middleware. Low
O4 OpenTelemetry tracing SDK wiring + collector config. Medium
O5 Grafana starter dashboards JSON dashboards committed to repo; auto-imported. Low
O6 Prism Senate Quality Dashboard panels Frontend work; depends on Prism bootstrap. Low (in Prism)
O7 Alert rules + paging integration Connects to existing on-call system. Low
O8 Privacy scrubber Reviewed against current log inventory. Medium (regression risk if it strips needed data)

Cadence: O1-O3 in week 1. O4 week 2. O5-O7 weeks 3-4. O8 alongside.

12. Work Log

2026-05-14 — task:14978bf3 — O1-O8 migration phases shipped

O1 (Structured logging + request_id): Already complete prior to this task — src/scidex_substrate/api/logging.py + RequestLogMiddleware.

O2 (Prometheus /metrics + domain counters/histograms): Added to metrics.py:

  • scidex_verb_calls_total{verb,type,outcome} — incremented from app.py verb dispatcher

  • scidex_verb_duration_seconds{verb,type} — histogram, 5ms–10s buckets, from dispatcher

  • scidex_validation_failures_total{verb,type,validator,path_prefix} — stub exposed for validators to call

  • scidex_version_conflicts_total{type,mergeable} — from app.py on SubstrateException(version_conflict)

  • scidex_rate_limit_rejections_total{scope,kind} — from rate_limit.py rejection branch

  • scidex_events_published_total{event_type} / scidex_events_delivered_total{event_type} — stubs for event bus / SSE to call

O3 (Bucket/quota/budget gauges): Added to metrics.py:

  • scidex_subscribe_active_consumers — consumers with last_seen_at within 5 min

  • scidex_subscribe_lag_events{consumer_id} — max(events.seq) - last_seen_seq per consumer

  • scidex_embed_queue_depth — count(embed_queue WHERE status='pending')

  • scidex_rate_limit_bucket_tokens{agent_id,scope} — from in-process RateLimiter.bucket_snapshot() (agent scope only, cap 100)

O4 (OpenTelemetry tracing): New file src/scidex_substrate/api/tracing.py:

  • Feature-flagged via SCIDEX_OBS_TRACING_ENABLED=1 (default off)

  • Lazy SDK init on first span; OTLP/gRPC to SCIDEX_OTEL_ENDPOINT

  • start_verb_span() / start_child_span() / record_error() public API

  • Head-based sampling at SCIDEX_OTEL_SAMPLE_RATE% (default 10%)

  • Optional dep group otel added to pyproject.toml

O5 (Grafana starter dashboard): monitoring/grafana/substrate-overview.json:

  • Panels: Overview (call rate, error rate, p50/p99 latency), By Verb, Validation, Subscribe Health, Embed Queue, Rate Limiting, Artifact counts.

O6 (Prism Senate Quality Dashboard): Prism frontend work — out of scope for this PR. Deferred to a Prism task.

O7 (Alert rules): monitoring/alerts.yaml:

  • SubstrateDown, Verb5xxBurst, LatencyP99Regress, ValidationStorm, ConflictStorm, EmbedQueueBacklog, SubscribeConsumerStuck, ActorRunaway

O8 (Privacy scrubber): Added to src/scidex_substrate/api/logging.py:

  • ScrubFilter log filter (feature-flagged via SCIDEX_OBS_SCRUBBER_ENABLED=1, on by default)

  • Strips JWT tokens, API keys, Bearer values, password=/secret= query params

  • Truncates log field strings at 200 chars per §10 spec

2026-05-15 — task:14978bf3 — rebase + cleanup verification

Rebased branch onto origin/main (02205620), dropped unrelated collider_resolve.py and schema_registry/init.py changes from prior commit. Diff now contains exactly 9 files: spec update, 5 Python modules (logging.py, metrics.py, tracing.py, app.py, rate_limit.py), pyproject.toml (otel dep group), monitoring/alerts.yaml, and the Grafana dashboard JSON. All phases O1-O8 verified present. Python syntax check passed.

13. Open questions

  1. Trace storage costs. 10% sampling at scale could still be a lot. Revisit if ingest blows up.

  2. Cardinality budget for metrics. agent_id label is high-cardinality (every agent). Use it sparingly; aggregate where possible.

  3. Audit log retention. db_write_journal grows linearly with writes. Plan for archival to cold storage at ~90 days.

  4. Prism vs Grafana split. Some dashboards are governance (Prism); some are ops (Grafana). Boundary: anything an agent or non-engineer needs to see is Prism.

  5. Existing monitoring/ folder at /home/ubuntu/monitoring — what’s there today? Likely partial; integrate with rather than replace.