SPEC-016 — Orchestra Integration

Field	Value
Status	Draft v1
Owner	kris.ganjam@gmail.com
Date	2026-04-29
Depends on	SPEC-001, SPEC-014
Pillar	Cross-cutting

TL;DR

scidex-substrate and scidex-prism become Orchestra-managed repos so spec maintenance, build work, and parity tracking happen through the same recurring-quest + one-shot-task pattern that currently drives v1 scidex. Each repo gets .orchestra-worktrees/task-<id>/ worktree convention, an AGENTS.md, a guard-main-writes.sh PreToolUse hook, an .claude/mcp.json, and one or two initial recurring quests. Convention follows v1 scidex, scope is narrower (specs, builds, parity tracking — not 5-layer self-improvement until v2 substrate is live).

1. Why Orchestra-managed during build

Two reasons:

1. Spec maintenance — specs live in scidex-substrate now (per SPEC-014). They evolve as we build. Orchestra agents (interactive Claude + autonomous workers) need a sanctioned way to edit them: worktree, PR, post-merge guard.

2. Build velocity — implementation PRs (PR 1a-12 in SPEC-001) can be parallelized across multiple agent sessions. Orchestra’s task fleet handles dispatch, capability matching, review-gate, post-merge restart. Reusing it skips reinventing those.

Once v2 is live, Orchestra runs the full self-improvement loop on v2 (5 layers, recurring quests for each). Until then, Orchestra runs a narrow set of meta-quests in the new repos.

2. Per-repo files

2.1 AGENTS.md

Tailored, shorter than v1 scidex’s AGENTS.md (~1000 lines). Substrate’s covers:

• Read this first; mission; SPEC-001 architecture pointer.

• Worktree discipline (never edit main; use .orchestra-worktrees/task-<id>/).

• Orchestra CLI workflow (orchestra task create --project SciDEX-Substrate ...).

• PR conventions ([Layer] short title [task:TASK_ID]).

• Self-hosted CI per SPEC-012 (no GitHub Actions).

• Test conventions per SPEC-007.

• Spec maintenance: specs are the source of truth; updates go through PR.

• Parity tracking via SPEC-015.

Prism’s AGENTS.md is similar but tailored to SvelteKit conventions (see SPEC-003).

2.2 .claude/hooks/guard-main-writes.sh

Same pattern as v1: blocks Write/Edit/Bash to the main checkout’s path; allows worktree paths and .claude/ config. Override via SCIDEX_ALLOW_MAIN_WRITE=1. Path-adapted per repo (substrate’s MAIN_DIR=legacy SciDEX repo-substrate).

2.3 .claude/settings.json

Registers the PreToolUse hook. Same shape as v1 scidex’s.

2.4 .claude/mcp.json

Stub initially (no substrate-side MCP server live during build). Once PR 2 ships the substrate MCP server, this points to python -m scidex_substrate.skill.mcp_server.

2.5 .gitignore additions

/.orchestra-worktrees/ and /.claude-slot.json ignored (worktree clutter shouldn’t track in repo).

2.6 .gitmodules

Substrate carries the same submodule structure as v1 scidex (see SPEC-014 §3.5):

• vendor/mimeo

• vendor/mimeographs

• vendor/kdense-skills (source of imported skills per SPEC-013)

• data/scidex-artifacts (optional Forgejo-backed reference checkout; not a v2 write target)

• data/scidex-papers

3. Orchestra side

The Orchestra fleet learns about the new projects:

• --project SciDEX-Substrate and --project SciDEX-Prism registered in Orchestra config.

• Worktree base paths configured.

• Per-project capability profiles (typically: coding>=7, reasoning>=6 for substrate; coding>=6 for prism).

This is configuration on the Orchestra side, not in the substrate/prism repos. Operator action; not spec’d here.

4. Initial recurring quests

4.1 Substrate

Quest	Frequency	Purpose
[Spec] Maintain v2 specs	every-12h	Walk specs in docs/design/; flag drift between code and spec; spawn one-shot tasks to reconcile
[Build] PR 1a-12 dispatch	every-6h	Read SPEC-001 §16 PR plan; identify the next unscheduled PR; spawn one-shot task to start
[Parity] Inventory curation	every-24h	Walk parity_target artifacts (per SPEC-015); identify uncategorized targets; spawn one-shot tasks to triage
[CI] Self-hosted runner health	every-6h	Check the substrate-ci.service systemd unit per SPEC-012; report failures

4.2 Prism

Quest	Frequency	Purpose
[Spec] Maintain Prism spec	every-24h	Watch SPEC-003 for changes; flag implementation drift
[Page] Tier-1 coverage	every-12h	Walk SPEC-003 §10.1 Tier-1 page list; identify next page to implement; spawn one-shot
[Component] Polymorphic build-out	every-12h	Build out polymorphic islands per SPEC-003 §4.2
[CI] Build health	every-6h	Check pnpm build output; report failures

These are seed quests. Operators tune cadence and scope as build progresses.

5. Worktree pattern

Same as v1 scidex (the convention is mature):

git -C legacy SciDEX repo-substrate worktree add \
  .orchestra-worktrees/task-<id> -b session/task-<id> origin/main

Orchestra workers are assigned a worktree at slot launch. Interactive Claude sessions derive worktree names from session id (per v1 scidex AGENTS.md).

6. Service restart hooks

Substrate has different restart-sensitive files than v1 scidex:

File	Service to restart
src/scidex_substrate/api/app.py	scidex-substrate-api.service
src/scidex_substrate/skill/mcp_server.py	scidex-substrate-mcp.service
src/scidex_substrate/migrations/*.sql	Run migration first; then restart api

Prism:

File	Service to restart
src/lib/components/*	none — Vite hot reload in dev; rebuild on prod deploy
src/hooks.server.ts	scidex-prism.service (after pnpm build)
vite.config.ts, svelte.config.js	rebuild + restart

7. Migration of v1 scidex AGENTS.md content

v1 scidex AGENTS.md has accumulated context across many incidents:

• Mermaid wiki bulk-edit history → substrate doesn’t have wiki bulk edits as a concept yet

• 30-second pull_main.sh reset → not present in substrate

• Forge conda environments → replaced by SPEC-002 pluggable runtimes

• 22-concurrent-workers PG contention → carries over but with different concrete tunings

The substrate’s AGENTS.md starts fresh. Incident-specific guidance accumulates as substrate-specific incidents happen.

8. Boundary between SPEC-012 (CI) and SPEC-016 (Orchestra)

CI (SPEC-012) tests pre-merge correctness. Orchestra (SPEC-016) dispatches build work. Different concerns:

Concern	SPEC-012 (CI)	SPEC-016 (Orchestra)
Trigger	git push to PR branch	recurring quest fires every N hours
Scope	one PR’s changes	task spec
Output	PR status (green/red)	task completion notes
Runner	systemd unit polling for new commits	systemd unit dispatching to slot agents

Both use self-hosted infra on the SciDEX VPC; both use gh CLI for status reporting.

9. Migration to v2-only Orchestra

After cutover (SPEC-006 Phase D):

• Orchestra’s --project SciDEX config retires (legacy).

• --project SciDEX-Substrate becomes the primary project for backend work.

• The 5-layer self-improvement quests from v1 are recreated on the substrate side (now using v2 verbs).

• Spec-maintenance quests stay on substrate.

9.1 Timing

Two-stage migration relative to SPEC-006 Phase C:

1. T-72h (pre-cutover): Orchestra fleet suspends per SPEC-006 §8. No new v1 tasks are claimed; in-flight tasks drain or are requeued. Operator audits open v1 tasks for disposition (§9.2).

2. Phase D day 1 (T+4h post-cutover): Once v2 smoke tests pass and traffic is open, the operator runs the activation runbook (§9.3) to bring the full v2 quest catalog live.

Starting pillar quests at Phase D day 1 (not Phase E) means the self-improvement loop runs during the stabilization window and surfaces regressions in v2 behavior early.

9.2 Pre-cutover prep (T-72h)

1. Audit all open v1 tasks:

   orchestra list --project SciDEX --status open
   

2. Triage each open task:

   • v1-specific tasks (wiki bulk edits, v1 actor migrations, v1 Forge conda-env management): close with orchestra complete --result pass --notes "superseded by v2 cutover".

   • v2-portable tasks (spec drift, parity gaps, knowledge-graph enrichment, Senate validator gaps): migrate to the substrate project via orchestra update --id <task_id> --project SciDEX-Substrate, or requeue with a note pointing to the equivalent SPEC-073 pillar quest.

3. Pause the v1 quest fleet, or the whole fleet if cutover requires a global quiet period. Project pause prevents new claims for that project only; global pause makes every project effectively paused while preserving each project’s own agent_state.

   orchestra global pause
   orchestra global status
   

   Expected global verification shows global_pause: paused and effective: paused for all projects. If only v1 must pause, use the Orchestra project pause mechanism and verify with the web/API state.

4. Confirm no agents are mid-claim on v1 work and that global pause is not still allowing get-next claimers:

   orchestra list --project SciDEX --status running
   ps -eo pid,ppid,user,stat,etime,comm,args \
     | rg 'orchestra.agent|orchestra get-next|orchestra supervisor|orchestra_watchdog' \
     | rg -v 'rg|codex'
   

5. For a global cutover drain, follow Orchestra global pause and graceful drain: stop the runtime timers, let live agents finish, and reap only stale DB-only leases that have no matching live agent process.

9.3 Post-cutover activation runbook (Phase D day 1)

Run after Phase C smoke tests pass (SPEC-006 Phase C step T+3.5).

Automated script (recommended):

# --dry-run prints commands without executing orchestra writes.
./scripts/ops/activate_v2_orchestra.sh [--dry-run]

Manual steps (if running by hand):

# 1. Confirm v2 substrate is live.
python scripts/ops/check_substrate_health.py   # exits 0 = healthy

# 2. Create all pillar + operational quests (SPEC-073 §4 + §5).
#    import_specs.py is idempotent — safe to re-run if quests were seeded during build.
python scripts/import_specs.py quests --apply --project SciDEX-Substrate

# 3. Activate the quest fleet.
orchestra quest activate --project SciDEX-Substrate --all

# 4. Archive the legacy SciDEX project so no new v1 work is dispatched.
orchestra project archive --name SciDEX

# 5. Verify state.
orchestra list --project SciDEX-Substrate --status open | head -20
orchestra stats --project SciDEX-Substrate
orchestra project show --name SciDEX   # must show status=archived

9.4 v1 → v2 quest mapping

v1 SciDEX ran a 5-layer self-improvement loop (51 quests across Atlas, Forge, Senate, Agora, Exchange). v2 adds an explicit Cross-cutting pillar, making it 6 pillars.

Structural change. v1 used fine-grained quests as the unit of dispatched work (one quest per feature area). v2 uses two tiers: coarse-grained pillar quests that aggregate work (SPEC-073 §5) plus fine-grained tasks per spec deliverable (SPEC-073 §6). The 51 v1 quests map to 6 pillar quests + 6 operational quests (SPEC-073 §4–§5) with 366 tasks distributed across them. This lets the pillar quests surface stalled work and match worker capabilities, while tasks remain PR-sized units.

Layer-to-pillar mapping. 39 of the 51 v1 quests have corresponding specs in the substrate (SPEC-074–113). The remaining 12 were v1-specific operational or housekeeping quests (mermaid wiki bulk-edit, conda-env management, legacy actor migrations) superseded by v2 architecture (SPEC-002 pluggable runtimes, SPEC-013 skills-as-artifacts) and dropped at cutover.

v1 quest layer	v1 quests	v2 pillar quest (SPEC-073 §5)	v2 specs (SPEC-073 §3)	Notes
[Atlas] Knowledge graph & corpus	6 (SPEC-074–079)	[Atlas] Knowledge graph, corpus, search	12 + 6: 005, 023, 027–029, 037, 044, 047, 052, 058, 066–067, 074–079	scidex.get/scidex.search are now typed functions (SPEC-001)
[Forge] Tools & runtimes	5 (SPEC-080–084)	[Forge] Runtimes, tools, skills, agent fleet	6 + 5: 002, 013, 020, 045, 046, 048, 080–084	Forge runtime model changes per SPEC-002; tools registered as typed artifacts via scidex.create
[Senate] Governance & quality	10 (SPEC-085–094)	[Senate] Governance, validators, quality	9 + 10: 008, 024, 031, 053, 055–056, 059, 068–069, 085–094	Validator artifacts per SPEC-031; votes via polymorphic scidex.signal
[Agora] Debates & social	3 (SPEC-095–097)	[Agora] Debates, social, comments	5 + 3: 030, 033, 038, 054, 067, 095–097	Debate artifacts per SPEC-045; comments via scidex.comment
[Exchange] Economics & markets	6 (SPEC-099–104)	[Exchange] Economics, faucet, market	8 + 6: 018, 022, 025–026, 032, 035–036, 060, 099–104	Faucet + collider per SPEC-025/032
(implicit in v1 infra)	9 (SPEC-105–113)	[Cross-cutting] Substrate core, schema, transport	21 + 9: 001, 004, 006–007, 009–011, 014–015, 017, 019, 021, 043, 049, 051, 061–065, 072, 105–113	New explicit pillar; covers schema registry, transport, migrations, LLM routing, causal mechanisms, epistemic tiers, universal search, artifact discussions, proposal generation, pre-registration, demo

The detailed per-spec pillar mapping is in SPEC-073 §3.

On v2 verbs: v1 quests used ad-hoc API calls against v1 endpoints. v2 quests interact via the MCP server (generated from @verb decorators per SPEC-001 §8); agents that previously called POST /api/wiki/edit now call the scidex.update MCP tool. The self-improvement prompt templates need updating at cutover to reference MCP tool names, not HTTP paths.

Operational quests ([Spec] Maintain v2 specs, [Build] Spec PR dispatch, [Parity] Inventory curation, [CI] Self-hosted runner health, [Migrations] Drift watch, [Schema] Registry hygiene) are already scoped to the substrate project from build-phase initialization (SPEC-073 §4). No re-creation needed; they just need activation at step 3 above.

9.5 Verification gate

Before declaring the migration complete (entry criteria for Phase E per SPEC-006 §2):

• orchestra stats --project SciDEX-Substrate shows all 6 pillar quests + 6 operational quests active.

• orchestra list --project SciDEX --status open returns empty (no open v1 tasks).

• orchestra project show --name SciDEX shows status=archived.

• First pillar quest fire (within 6h of activation) produces at least one claimed task in the substrate project — confirms fleet dispatch is routing to v2.

• No v1-path tools or endpoints referenced in the newly-active quest prompts (audit via orchestra quest show --all | grep "/api/").

Failure on any gate item: hold Phase E until resolved. Do not archive the v1 systemd service (SPEC-006 §2 Phase E) while any quest or task still targets v1 infrastructure.

Verification — 2026-05-15T04:58:00Z

Result: FAIL Verified by: minimax:31 via task 1402ccc2-c164-4803-9fc3-b29b567634f4

Tests run

Target	Command	Expected	Actual	Pass?
Gate 1: 6 pillar + 6 operational quests active	MCP list_tasks(project=SciDEX-Substrate, task_type=recurring)	12 recurring quests	0 recurring quests in SciDEX-Substrate; 19 recurring quests belong to v1 SciDEX project	✗
Gate 2: v1 task queue empty	MCP list_tasks(project=SciDEX, status=open) — sampled first 20	0 tasks	20 open v1 tasks (recurring CI/Senate/Exchange/Arenas quests); ~100 more likely remain given prior count of 124	✗
Gate 3: SciDEX archived	MCP get_task + CLI orchestra project show --name SciDEX	status=archived	DB symlink /home/ubuntu/Orchestra/orchestra.db → /data/orchestra/orchestra.db broken (target dir missing); MCP returns unable to open database file	✗ (DB unavailable)
Gate 4: First pillar quest fire	Quest must exist before it can fire	Quest fires → claimed task	No pillar quests exist in SciDEX-Substrate to fire	✗ (N/A — prerequisite missing)
Gate 5: No v1-paths in quest prompts	Cannot audit without accessible quest list	No v1 API paths	Quest catalog inaccessible due to DB symlink failure	✗ (DB unavailable)

Additional checks

Check	Command	Result
Substrate health	python scripts/ops/check_substrate_health.py --host 127.0.0.1 --port 8000	FAIL — 2 pending migrations (0149_spec110_proposals.sql, 0148_spec112_preregistration.sql); /health returns 200 but /readyz returns 404
DB reachability	psycopg to scidex_v2	OK — 704 public tables reachable
Quest creation script	ls scripts/ops/activate_v2_orchestra.sh	Present + executable; scripts/import_specs.py has --project flag (added by PR #962)
Recent main activity	git log --oneline origin/main -15	4 PRs merged since previous verification: SPEC-113 screenshot bot (#960), SPEC-002 runtime handlers (#968), SPEC-016 §9.3 activation runbook (#962), SPEC-016 §9.4 v1→v2 quest mapping (#967)

Attribution

The current failing state reflects the system before Phase D activation has been performed:

• SciDEX-Substrate project exists (task rows confirm project_id scidex-substrate), but the quest catalog has not been seeded via scripts/import_specs.py quests --apply.

• The activate_v2_orchestra.sh script (PR #962, 2026-05-14 21:56) provides the automation to run the activation steps, but it has not yet been executed.

• v1 SciDEX project is still fully operational with 19 active recurring quests (CI, Senate, Exchange, Arenas, Atlas, Forge pillars) and approximately 100 open one-shot tasks — no cutover has occurred.

• DB availability issue: orchestra CLI binary cannot access its database via symlink (/data/orchestra/ does not exist); MCP server accesses DB through a different path (HTTP fallback to port 5000 or UNIX socket) and works for list_tasks calls.

Notes

• Phase D activation not yet run: scripts/ops/activate_v2_orchestra.sh exists and is executable; running it would seed the quest catalog, activate the quest fleet, and archive v1. This is the prerequisite gate to unlock §9.5.

• Migration gap: Substrate has 2 pending migrations (0149, 0148) that need to be applied before /readyz returns 200 — this may block the activation script’s Step 1 health check.

• SPEC-073 context: Per SPEC-073 §4, the 12 recurring quests (6 pillar + 6 operational) should be created via scripts/import_specs.py quests --apply --project SciDEX-Substrate. This step must run before Gate 1 can pass.

• Gate ordering: Gates 2–5 are all blocked by the same root cause — Phase D activation (SPEC-016 §9.3) has not been performed. The activation script is the path forward.

Verification — 2026-05-14T21:50:00Z

Result: FAIL Verified by: minimax:31 via task 1402ccc2-c164-4803-9fc3-b29b567634f4

Tests run

Target	Command	Expected	Actual	Pass?
Gate 1: 6 pillar + 6 operational quests active	orchestra quest list --project SciDEX-Substrate via MCP list_tasks (freq≠null)	12 recurring quests	0 recurring quests found; only 1 quest e83d96fc-62c exists with 6 running non-recurring tasks	✗
Gate 2: v1 task queue empty	orchestra list --project SciDEX --status open via MCP	0 tasks	124 open v1 SciDEX tasks remain	✗
Gate 3: SciDEX archived	orchestra project show --name SciDEX via MCP	status=archived	MCP returns HTTP 404: Task 'SciDEX' not found — project lookup not task lookup; orchestra CLI fails with unable to open database file (symlink /home/ubuntu/Orchestra/orchestra.db → /data/orchestra/orchestra.db broken)	✗ (DB unavailable)
Gate 4: First pillar quest fire	Quest must exist before it can fire	Quest fires → claimed task	No pillar quests exist to fire	✗ (N/A — prerequisite missing)
Gate 5: No v1-paths in quest prompts	orchestra quest show --all | grep "/api/"	No v1 API paths	Cannot run — quest list inaccessible (DB unavailable)	✗ (DB unavailable)

Attribution

The current failing state reflects the system before migration has started:

• SciDEX-Substrate project exists (confirmed by task rows with project_name: SciDEX-Substrate), but no recurring quests have been seeded.

• The single quest e83d96fc-62c is likely a “Cross-cutting” pillar catch-all quest created by recent import_specs runs.

• v1 SciDEX project is still fully operational with 124 open tasks — no cutover has occurred.

Notes

• DB availability issue: The orchestra CLI binary cannot connect to its database (unable to open database file — symlink to /data/orchestra/orchestra.db is broken). The MCP server (orchestra.mcp) accesses the DB through a different path (via UNIX socket or HTTP fallback to the web service at port 5000) and works correctly.

• Quests vs. tasks: The MCP list_tasks call correctly returns tasks but list_tasks with task_type=recurring returns zero results because the DB is structured as tasks, not quests. The quest catalog is managed separately.

• Pre-cutover state: SPEC-006 Phase C (Cutover) has not occurred. This verification gate is defined as an entry criterion for Phase E, which cannot begin until Phase C completes. The gate is inherently blocked until pre-cutover preparation (SPEC-016 §9.2) is performed.

• SPEC-073 context: Per SPEC-073, the quest catalog is seeded via scripts/import_specs.py quests --apply --project SciDEX-Substrate. This has not been executed (or its results are not yet reflected in the live quest registry). The gate cannot pass until that script runs and the resulting quests activate.

9.6 Verification status reconciliation - 2026-05-18

The historical verification tables in §9.5 show pre-cutover and CLI-access failures. SPEC-073 §13 is the newer runbook execution log and reports 366 task rows with task creation/backfill partially complete. Treat §9.5 as historical gate evidence and SPEC-073 §13-§14 as the current source for task-catalog status until a fresh Orchestra host-side verification replaces both.

10. Open questions

1. Naming consistency: project name matches repo name (SciDEX-Substrate, SciDEX-Prism)? Or use shorter aliases? Default: match repo name; aliases for CLI ergonomics later.

2. Cross-repo coordination: a substrate PR that needs a Prism update (e.g., new schema field for a TS type). How does Orchestra link the two? Probably: substrate PR carries [link:prism-pr#N] footer; reviewer verifies both before merge.

3. Read-only v1 access: should Orchestra agents working in substrate be able to read v1 api.py for reference? Yes — read-only mount; v1 is reference per SPEC-014.

4. Initial quest population: do the seed quests in §4 get created automatically at first deploy, or does the operator run them manually? Default: operator runs once at deploy.

12. Work Log

2026-05-15 — §9.2 pre-cutover prep automation [task:8e0a728f-c48e-4d29-8114-6752942a6fd6]

What: Created scripts/ops/pre_cutover_prep.py to automate the four §9.2 steps.

Audit snapshot (2026-05-15, dry-run):

• v1 SciDEX open tasks: 124

• v1-specific (will close at T-72h): 10

  • [Demo], [UI], [Quality] showcase/forge-playground/spotlight-notebook tasks

• v2-portable (will note + migrate at T-72h): 114

  • All [Senate], [Exchange], [Agora], [Atlas], [Forge] core, [Gaps], [Arenas] tasks

• Running v1 agents: 0

Notes on step 3 (pause): The orchestra quest CLI in this deploy does not expose pause-all. The script uses POST /api/projects/SciDEX/pause via the HTTP API instead (requires ORCHESTRA_WEB_TOKEN). This is equivalent: it quiesces the project so no new tasks are dispatched.

Notes on task migration: PATCH /api/tasks/{id} does not expose a project field. v2-portable tasks are instead completed with a note pointing to the SPEC-073 pillar quest that owns the equivalent work in SciDEX-Substrate. The operator then activates those pillar quests at T+4h per §9.3.

Status at time of this log: v2-rc-checklist shows 0% across all M1..M7 sections; cutover is not imminent. The automation is ready to run; execute only at actual T-72h.

11. Interaction with other specs

• SPEC-001: unchanged.

• SPEC-006: cutover archives v1 scidex; Orchestra config flips to substrate-primary. §9 migration runbook keys off SPEC-006 Phase C/D timing.

• SPEC-012: CI runs alongside Orchestra; different concerns.

• SPEC-013: substrate’s native skills include scidex onboarding (live now).

• SPEC-014: submodule list referenced here.

• SPEC-015: parity inventory curation is one of the recurring quests.

• SPEC-073: expands the seed quest list in §4.1 into the full pillar + operational catalog; §9 migration runbook activates those quests. SPEC-073 §9 (apply order) is the canonical operator procedure for initial quest creation; this §9 covers the post-cutover activation step.