Agent Developer Guide

Worktree workflow, PR layer prefixes, multi-worker .pth footgun + workarounds, per-task spec discipline.

Source: docs/tutorials/agent-developer-guide.md

Agent developer guide

This document is for AI agents (Claude, Codex, Orchestra workers) doing real work in this repo. It encodes the conventions from AGENTS.md — the worktree discipline, PR layer prefixes, multi-worker install footgun, test patterns, and per-task spec policy — in one place, with concrete commands. Human contributors should also read this if they’re going to work alongside agents.

Read AGENTS.md first. This guide expands the parts of it most agents trip on, with worked examples.

1. Worktree discipline (mandatory)

You may not edit files on the main checkout. A PreToolUse hook at .claude/hooks/guard-main-writes.sh blocks Write, Edit, and any Bash that targets /home/ubuntu/scidex-substrate directly. The override ! export SCIDEX_ALLOW_MAIN_WRITE=1 exists for emergencies and should not appear in normal task work.

Always work in a worktree:

git -C /home/ubuntu/scidex-substrate worktree add \
    .orchestra-worktrees/task-<id> \
    -b session/task-<id> \
    origin/main

cd /home/ubuntu/scidex-substrate/.orchestra-worktrees/task-<id>
# ... edit, commit, push to origin/session/task-<id> ...

For Orchestra-managed sessions, the supervisor pre-creates the worktree and binds it before launch — you’ll find yourself already inside .orchestra-worktrees/task-<id> on first prompt.

When done, the worktree is not automatically cleaned up. The supervisor handles GC. Don’t git worktree remove from inside an agent session unless you’ve moved out of the directory first.

2. PR layer prefixes

Every PR title is [Layer] short imperative summary [task:TASK_ID], where Layer is one of:

Prefix Touches
[Skill] Verbs, framework, validators, codegen, MCP schema, write metering
[Core] DB pool, auth, events, identity, errors, retries
[Schema] schema_registry/, default_schemas/*.json, migrators
[API] FastAPI routes, SSE handler, middleware, REST controllers
[MCP] MCP server, tool registration
[Migration] SQL migrations + Python transformers
[Tools] Tool artifacts (tools/), runtime handlers
[Skills] Skill artifacts (skills/), importers
[Tests] Test infrastructure or new tests at any layer
[Docs] Specs, READMEs, AGENTS.md edits, tutorials
[CI] Hooks, scripts/ci/, runner config
[Parity] parity_target artifacts, v1-parity verification

PR body: 1-3 bullet Summary, then Test plan checklist. Cite the SPEC section the change implements (SPEC-001 §7.2, SPEC-013 §3). Don’t ship code that diverges from a spec; either update the spec first or rewrite the change to match.

Footer for AI-generated PRs (set automatically by Claude Code):

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Squash-merge on green CI: gh pr merge --squash --delete-branch.

3. The multi-worker .pth footgun

This bites at least once per /batch wave. The full incident write-up is in AGENTS.md §Multi-worker concurrent install footgun. Short version:

When multiple workers each clone the repo to /tmp/m<N>-<unit>-<rand>/substrate and each run pip install -e ".[dev]", they share the host’s miniconda site-packages. Each editable install rewrites ~/miniconda3/.../site-packages/__editable__.scidex_substrate-0.0.1.pth to point at the most recent install path. So worker B’s tests can import worker A’s stale code at random — manifesting as “module not found” on files the worker just wrote, or test assertions failing on code the worker never wrote.

Three workarounds, increasing isolation:

# 1. PYTHONPATH pin — cheapest, no install needed for pure pytest.
PYTHONPATH=$PWD/src pytest tests/unit/skill/verbs/test_my_verb.py

# 2. Force-reinstall before each checkpoint.
pip install -e . --no-deps --force-reinstall

# 3. Per-worker venv — cleanest, recommended for /batch waves.
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Default to (1) for fast iteration; switch to (3) if your unit needs multiple editable packages or you’re spawning parallel pytest runs.

4. The pytest DSN guard

tests/conftest.py refuses to run pytest against the live scidex or scidex_v2 databases, or any database whose name doesn’t match the *[_-]test[_-]* pattern. SCIDEX_TEST_DSN_ALLOW_PROD is intentionally ignored — there is no escape hatch.

The guard reads SCIDEX_TEST_DSN first, falling back to SCIDEX_DSN. The discipline:

# Per-task test database (recommended for /batch waves).
createdb scidex_v2_test_task_$TASK_ID
export SCIDEX_TEST_DSN=postgresql:///scidex_v2_test_task_$TASK_ID
pytest tests/unit

If pytest exits with Refusing to run tests against non-test SciDEX database, that’s the guard talking. Rename the database to end in _test and re-run.

For unit tests that don’t touch the DB (most of tests/unit/), the DSN isn’t read — the suite runs against module imports only. The guard still fires when the env var is set, so it costs nothing to keep it set always.

5. Test patterns

Per SPEC-007 the substrate has six test layers; the three you usually write are unit, service, and integration. See writing-a-verb.md for the full walked example.

The minimum any new verb owes:

Layer Lives under Wraps Catches
unit tests/unit/skill/verbs/test_<verb>.py Nothing — pure model checks Pydantic model coercion
service tests/service/skill/verbs/test_<verb>.py BEGIN; … ROLLBACK; on a test DB Verb body + side effects in journal/events
integration tests/integration/api/test_<verb>_http.py httpx.AsyncClient against the FastAPI app Auth, serialization, status codes

The pre-push hook runs pytest tests/unit only. The self-hosted CI runner runs service + integration. Skip the unit hook only with SCIDEX_SKIP_PREPUSH=1 and have a specific reason.

6. Per-task spec policy

Every concrete code task — forge adapters, skills, verbs, tests, runtime, schema, migrations — MUST be registered with --spec docs/planning/specs/<short_slug>_spec.md, where the file exists and is committed/pushed before orchestra task create is called. --no-spec is permitted only for genuinely no-repo operational tasks (a verified no-op, a pure ops sweep).

This is the rule from AGENTS.md §Task generation and the entry in the user’s memory at feedback_generated_tasks_require_specs.md. Specs in docs/planning/specs/ give the executing agent an acceptance contract and a validation path. Without that contract, the work is unaccountable.

A spec file has frontmatter + five sections:

---
task_id: <uuid-from-orchestra-task-create-output>
title: "[Skill] Add scidex.paper.cite_count verb"
type: one_shot
spec_path: docs/planning/specs/paper_cite_count_verb_spec.md
---

## Goal

Add a `scidex.paper.cite_count` verb that returns the number of
inbound `evidence_for` / `evidence_against` links pointing at a
paper artifact.

## Evidence

- v1 `paper.citation_count` is a denormalized column that drifts
  from the live `artifact_links` table; reproduced drift in
  manual audit 2026-05-12.
- `src/scidex_substrate/skill/verbs/paper_figure_extract.py` is
  the only existing `scidex.paper.*` verb; pattern is established.
- SPEC-001 §7.2 mandates polymorphic verbs over per-type endpoints.

## Approach

1. Write input/output Pydantic models per writing-a-verb.md.
2. Decorate with `@verb(name='scidex.paper.cite_count',
   read=True, public=True)`.
3. SQL: COUNT(*) GROUP BY predicate on artifact_links filtered to
   to_type='paper' and to_id=<arg>.
4. Reject non-paper refs with validation_failed.
5. Write unit + service + HTTP tests per SPEC-007.

## Acceptance

- POST /api/scidex/paper/cite_count with {"ref": "paper:p-1"}
  returns 200 with {ref, total, by_predicate}.
- POST with a non-paper ref returns 400 with code=validation_failed.
- Unit, service, and HTTP tests pass.
- Verb appears in GET / verb list.

## Validation

- Pre-push: pytest tests/unit/skill/verbs/test_paper_cite_count.py
- Post-merge: hit the route on the dev substrate; confirm count.

The order is strict:

  1. Write and commit the spec file on a worktree branch.

  2. Push the branch (so the spec is reachable by URL).

  3. orchestra task create --spec docs/planning/specs/<file>.md ...

  4. Take the task and do the work in a fresh worktree.

Order matters because Orchestra’s queue links to the spec by path, and a missing spec at execution time is a hard failure.

Drift note from AGENTS.md: some historical done tasks have empty spec_path. That is drift, not license to use --no-spec. When the DB state and AGENTS.md disagree, AGENTS.md wins. Flag the drift in your task notes and move on.

7. Orchestra task workflow

The Orchestra MCP server is your primary task interface. From AGENTS.md §Orchestra integration:

# Claim next task assigned to your slot
orchestra task get-next --slot $ORCHESTRA_SLOT --project SciDEX-Substrate

# Long-running tasks — heartbeat every 5-10 minutes
orchestra task heartbeat --id $TASK_ID

# Completion paths (pick honestly):
# A — real work shipped: do nothing; supervisor watches the branch
#     and auto-completes on merge.
# B — verified no-op: orchestra complete --result pass \
#       --notes "no-op: <why>"
# C — cannot recover: orchestra requeue --id $TASK_ID
#     or: orchestra update --status blocked

Three completion paths. Pick the honest one. “Auto-complete on merge” means: ship a PR with [task:<id>] in the title, and the supervisor detects the merge and closes the task. You do not call orchestra complete for a real-work task.

The MCP server tool surface is the same — see the tools listed under mcp__orchestra__orchestra__* in the available tool list. For example mcp__orchestra__orchestra__claim_next_task, mcp__orchestra__orchestra__heartbeat, mcp__orchestra__orchestra__complete_task.

8. Pre-commit and pre-push hooks

Per SPEC-012, CI runs entirely on a self-hosted runner — no GitHub Actions. Locally:

./scripts/install-hooks.sh   # one-time per worktree
./scripts/ci.sh              # reproduce CI locally
Hook Runs Override (use sparingly)
pre-commit ruff format --check, ruff check SCIDEX_SKIP_HOOKS=1
pre-push ruff check, mypy, pytest tests/unit SCIDEX_SKIP_PREPUSH=1

If a hook fails, fix the underlying issue. Don’t --no-verify your way past a failing hook — that’s an explicit AGENTS.md violation unless the user asks for it. If pre-commit’s ruff check fails on a file you didn’t touch, you’ve imported it indirectly (e.g., the file was added in another worker’s .pth-shadowed install); use the multi-worker workarounds from §3.

9. Where to verify your work

Before opening a PR:

# 1. Pure model checks — sub-second.
PYTHONPATH=src pytest tests/unit -x

# 2. Lint + types — sub-second on a single file.
ruff check src/scidex_substrate/skill/verbs/my_new_verb.py
mypy src/scidex_substrate/skill/verbs/my_new_verb.py

# 3. Service tests against a _test DB (per-task DB recommended).
SCIDEX_TEST_DSN=postgresql:///scidex_v2_test_task_$TASK_ID \
    pytest tests/service/skill/verbs/test_my_new_verb.py -x

# 4. Integration against the running app.
SCIDEX_TEST_DSN=postgresql:///scidex_v2_test_task_$TASK_ID \
    pytest tests/integration/api/test_my_new_verb_http.py -x

# 5. Local end-to-end smoke.
PYTHONPATH=src python3 scripts/ops/verify_substrate_ephemeral.py
echo "exit=$?"

The ephemeral verifier spins up a throwaway Postgres, applies every migration, runs a curl smoke against the API, and tears down. Exit 0 means the full stack works against the current main. It is the same script the pre-push hook runs.

10. Standing pre-approvals for /batch and /loop

From AGENTS.md §Autonomous mode: when /batch or /loop is invoked, the coordinator runs autonomously without prompting for plan-mode approval. The act of invoking /batch IS the approval. Workers spawned during autonomous loops must not ask “may I proceed” — they do the work, ship the PR, and report. A worker that hits a genuine safety concern should report inline and continue with other work rather than blocking the whole loop.

The pre-approval covers the standard substrate dev workflow:

  • clone to /tmp/m<N>-...

  • gh CLI for PR operations

  • push to substrate branches

  • pytest, ruff, mypy

  • edit memory + plan files

What still requires explicit user input:

  • Pushes to v1 SciDEX main (frozen — never push there).

  • Any write to v1 Postgres scidex.

  • Force-push / reset --hard on shared branches.

  • --no-verify on commit hooks.

  • Anything outside substrate dev scope.

11. Self-checks for AI work

Before shipping a PR, run these mental checks:

  • Did I work on a worktree? If you can git status clean in /home/ubuntu/scidex-substrate, you bypassed the hook. Move your changes.

  • Did I commit a spec? Per §6, code work requires a spec under docs/planning/specs/ committed before the work landed.

  • Does the PR title have the right [Layer] prefix? §2.

  • Does the PR title carry [task:<id>]? Required for supervisor auto-complete on merge.

  • Are tests at all three layers in the diff? Unit + service + HTTP, per §5.

  • Do my error codes match _status_for_code in api/app.py? Invented codes return 500 by default — pick from the canonical set.

  • Did I avoid --no-verify? If a hook failed, the underlying issue isn’t fixed.

  • Does my change drift from a spec? If yes, update the spec first.

12. References