Scheduled content workers (SPEC-046)

Operator runbook for the seven LLM-driven secondary content loops that sit alongside the primary debate orchestrator (SPEC-045). Each is a oneshot systemd unit fired by its companion timer; each writes audit rows to scheduled_worker_runs (migration 0045).

Workers and cadences

Name	Cadence (UTC)	Daily $ cap	Output
notebook_debate_scheduler	Mon 06:00	$1 (no LLM)	pantheon_debate shells with methodology trio
cross_disease_analogy	Daily 06:15	$10	hypothesis artifacts (kind=cross_disease_analogy)
open_question_from_falsified	Daily 03:30	$4	knowledge_gap (kind=falsification_residual)
brief_writer_daily	Daily 07:00	reserved	agent_work_packet (kind=daily_brief_authoring)
daily_digest	Daily 09:00	reserved	agent_work_packet (kind=daily_digest_authoring)
comment_classifier	Every 5 min	reserved	agent_work_packet (kind=comment_classification)
persona_drift_daily	Daily 03:00	$2	wiki_page (kind=persona_drift)
sunday_championship	Sun 00:00	$5/wk	arena (kind=championship)

Total daily budget bucket: 25/day weekday, 30/day Sun (championship adds 5/week). Spec §2.4 says 30/day total; we sit at 25/day with 5/week peak — safely under.

Tuning budgets

Operator overrides via env vars in /etc/scidex-substrate/scheduled-workers.env. Pattern: SCIDEX_SCHED_<NAME>_DAILY_USD=<float>.

# /etc/scidex-substrate/scheduled-workers.env
ANTHROPIC_API_KEY=sk-ant-...
SCIDEX_SCHED_DAILY_DIGEST_DAILY_USD=10
SCIDEX_SCHED_BRIEF_WRITER_DAILY_MODEL=claude-sonnet-4-5-20250929

The EnvironmentFile= line in each unit is - prefixed so a missing file degrades to defaults rather than failing the unit.

Pausing a worker

Set the kill-switch env var to any falsey value:

sudo tee -a /etc/scidex-substrate/scheduled-workers.env <<'EOF'
SCIDEX_SCHED_BRIEF_WRITER_DAILY_ENABLED=false
EOF
sudo systemctl daemon-reload

The next timer fire will run, hit the kill-switch, write status='disabled' to scheduled_worker_runs, and exit cleanly. The operator can grep the audit table for status='disabled' to confirm the muzzle is in effect.

What each worker emits

• notebook_debate_scheduler: top-N notebooks by quality + objection count get a pantheon_debate shell with personas [methodologist, statistician, replicator]. SPEC-045 picks them up for round-by-round arguing.

• cross_disease_analogy: top-N high-quality hypotheses generate up to 3 analogy hypotheses each in DIFFERENT disease verticals; each cites the shared mechanism (gene/pathway/class).

• open_question_from_falsified: hypotheses with lifecycle_state='falsified' in last 7d get up to 2 successor research questions written as knowledge_gap artifacts.

• brief_writer_daily: selects the top active topics and source artifacts, then queues one agent-owned daily brief packet per topic. The agent runtime owns prose generation, review, and final wiki_page publication.

• daily_digest: computes the KPI strip, top movers, and resolved debate candidates, then queues one agent-owned digest authoring packet. The agent runtime owns prose generation, review, and final wiki_page publication.

• comment_classifier: finds unclassified comments and queues idempotent agent-owned classification packets. The agent runtime owns provider selection, adaptive context use, percolation_tag writes, and comment.classified follow-through.

• persona_drift_daily: persona-by-persona drift detection (>2σ on budget_usd or steps mean shift between current 7d vs prior 28d window). Calls Haiku once per drift event for the natural-language description.

• sunday_championship: one cross-domain arena artifact seeded with the top hypothesis from each disease vertical. SPEC-032’s scidex.arena.judge_pairing runs the actual Swiss rounds.

Operator commands

# List registered workers
python -m scidex_substrate.scheduled_workers list

# Dry-run any worker (no DB writes, no LLM cost)
python -m scidex_substrate.scheduled_workers daily_digest --dry-run --json

# Force-fire a worker on-demand
sudo systemctl start scidex-worker-daily-digest.service

# Tail journal for a worker
sudo journalctl -u scidex-worker-daily-digest.service -f

Audit + cost dashboard

Read recent runs via the verb (admin / system token required):

curl -X POST http://127.0.0.1:8200/api/scidex/scheduled_worker/runs/list \
  -H "Authorization: Bearer $CRON_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"since_hours": 24}'

Or with SQL:

-- Today's spend per worker
SELECT name,
       COUNT(*) AS runs,
       SUM(items_produced) AS items,
       ROUND(SUM(budget_usd_used)::numeric, 4) AS spend_usd
FROM scheduled_worker_runs
WHERE started_at::date = CURRENT_DATE
GROUP BY name
ORDER BY spend_usd DESC;

-- Stuck or errored runs in last 24h
SELECT id, name, started_at, status, error
FROM scheduled_worker_runs
WHERE status IN ('error', 'running')
  AND started_at > NOW() - INTERVAL '24 hours'
ORDER BY started_at DESC;

Escalation

Symptom	First check	Action
status='budget_exhausted' repeatedly	Confirm intentional (operator bumped a worker’s run frequency?)	Raise SCIDEX_SCHED_<NAME>_DAILY_USD in env file.
Same worker status='error' 3+ times in 24h	journalctl -u scidex-worker-<name> for stack trace	Page substrate maintainer; flip SCIDEX_SCHED_<NAME>_ENABLED=false while debugging.
LLM cost spike	SUM(budget_usd_used) GROUP BY name over last week	Check whether operator flipped a worker to Sonnet via _MODEL override.
Worker stuck status='running' for > 30m	systemctl status scidex-worker-<name>	Stop the unit; the next timer fire is independent.
No rows for a worker over 24h	systemctl list-timers scidex-worker-* to verify the timer is loaded	Re-enable the timer if the auto-installer dropped it (substrate #389).

Cross-worker rate-limit collisions

Per task brief: SPEC-048 territory. If two LLM-driven workers fire on the same minute and both burn through Anthropic rate-limit, file a follow-up against SPEC-048 watchdogs. The current timer offsets (03:00 / 03:30 / 06:00 / 06:15 / 07:00 / 09:00) keep things spaced.

Out-of-scope (deferred per spec §6)

These v1 jobs are NOT shipped: scidex-personal-weekly-digest, scidex-debates-weekly-snapshot, scidex-expert-review-queue-weekly, scidex-disease-priority, scidex-field-velocity-daily, the v1 persona daemon (operator-gated, indefinite hold).

Email distribution of the daily digest is also out of scope — v1’s sender lives in scripts/run_daily_digest.py and isn’t a substrate concern.