SPEC-007 — Test Strategy

Unit/integration/contract test layers, fixtures, and the *_test database guard.

Source: docs/design/spec-007-test-strategy.md

SPEC-007 — Test Strategy

Field Value
Status Draft v1
Owner kris.ganjam@gmail.com
Date 2026-04-28
Depends on SPEC-001, SPEC-006
Pillar Cross-cutting

TL;DR

Six test layers ensure the polymorphic substrate works as designed and stays working under change: unit (Pydantic models, validators), service (in-process verb calls), HTTP integration (FastAPI TestClient), MCP integration (real MCP server, real tool calls), agent acceptance (live Claude/Codex sessions), end-to-end (Prism + substrate). Plus three cross-cutting suites: schema-conformance, concurrent-locking, and migration tests. Performance benchmarks track regressions per PR.

1. Why six layers

The failure modes the substrate must defend against are heterogeneous:

Failure Caught by
Pydantic model coercion bug Unit tests
Verb dispatches to wrong service function Service-layer tests
FastAPI route doesn’t match Pydantic schema HTTP integration tests
MCP tool spec doesn’t match agent expectations MCP integration tests
Agent can’t actually use the tool to complete a task Agent acceptance tests
User can’t sign in and see their hypothesis End-to-end tests
Concurrent updates corrupt state Concurrent-locking tests
Schema change orphans existing artifacts Migration tests
New code regresses search latency 10× Performance benchmarks

No single layer covers them all. Six is the minimum cost.

2. Layer 1 — Unit tests

Per scidex/skill/verbs/<verb>.py and scidex/skill/validators/<validator>.py:

  • Pydantic input model accepts valid args; rejects invalid with field-level error.

  • Pydantic output model serializes deterministically.

  • Pure functions (validators, content-hash recipes) tested with table-driven cases.

  • No DB access. No I/O. Run in <1s for the full layer.

Goal: every new verb / validator / framework helper has a unit test. Coverage gate at 90%.

# tests/unit/skill/verbs/test_get.py
def test_get_in_rejects_unknown_type():
    with pytest.raises(ValidationError):
        GetIn(ref=Ref(type="nonexistent", id="x"))

def test_get_in_accepts_optional_version():
    args = GetIn(ref=Ref(type="hypothesis", id="h-1", version="sha256:abc"))
    assert args.ref.version == "sha256:abc"

3. Layer 2 — Service-layer tests

Per service function (scidex.skill.<verb>) called in-process against a transactional test DB.

  • Each test is wrapped in BEGIN; … ROLLBACK;. No persistent state.

  • Fixtures populate minimum data: one of each artifact type, one agent.

  • Tests cover happy path + each error code in scidex_errors.

  • Verifies side effects: rows in db_write_journal, events on events table.

# tests/service/skill/verbs/test_create.py
def test_create_hypothesis_stamps_identity(test_db, test_actor):
    ctx = test_context(actor=test_actor)
    out = create.handle(CreateIn(type="hypothesis", content={...}), ctx)
    journal = test_db.query("SELECT * FROM db_write_journal WHERE id = %s", out.write_id)
    assert journal.agent_id == test_actor.id

def test_create_hypothesis_rejects_invalid_content(test_db, test_actor):
    ctx = test_context(actor=test_actor)
    with pytest.raises(ValidationFailed) as exc:
        create.handle(CreateIn(type="hypothesis", content={"title": ""}), ctx)
    paths = [p["path"] for p in exc.value.problems]
    assert "/content/title" in paths

Goal: every verb has a service-layer test for happy path + every named error code.

4. Layer 3 — HTTP integration tests

fastapi.TestClient with a real (test) DB and JWT minted in setup.

  • Verifies HTTP status codes match the structured error model.

  • Verifies serialization round-trips (Pydantic → JSON → Pydantic).

  • Verifies auth: missing token → 401; insufficient permission → 403.

  • Verifies rate-limit middleware is active (one negative test per scope kind).

# tests/http/test_v2_create.py
def test_create_returns_201_with_envelope(client, jwt_contributor):
    resp = client.post("/api/scidex/create",
                       json={"type": "hypothesis", "content": {...}},
                       headers={"Authorization": f"Bearer {jwt_contributor}"})
    assert resp.status_code == 201
    assert resp.json()["ref"]["type"] == "hypothesis"

5. Layer 4 — MCP integration tests

Spin up the real MCP server in a subprocess. Send JSON-RPC over stdio. Verify:

  • Tool list matches the verb registry exactly.

  • Each tool’s input schema is valid JSON Schema Draft 2020-12.

  • Calling each tool with valid args succeeds.

  • Errors propagate as structured tool errors (not crashes).

# tests/mcp/test_tool_inventory.py
def test_skill_mcp_lists_all_verbs(mcp_subprocess):
    tools = mcp_subprocess.send({"method": "tools/list"})["tools"]
    expected = {"scidex.get", "scidex.list", "scidex.search", ...}  # full set
    assert {t["name"] for t in tools} >= expected

def test_scidex_get_via_mcp_returns_envelope(mcp_subprocess, fixture_hypothesis):
    result = mcp_subprocess.send({
        "method": "tools/call",
        "params": {"name": "scidex.get", "arguments": {"ref": str(fixture_hypothesis)}}
    })
    assert result["artifact"]["type"] == "hypothesis"

6. Layer 5 — Agent acceptance tests

This is what’s uniquely important for an agent-first substrate. Tests run a real Claude/Codex/Orchestra agent against a live (test) substrate and verify it can complete realistic tasks.

6.1 Test taxonomy

Task class Example
Find “Find the top 5 hypotheses about α-synuclein with mechanistic_plausibility ≥ 0.8”
Create “Propose a hypothesis for LRRK2 in Parkinson’s, citing 3 papers from 2024”
Update “Mark hypothesis h-7a9c as ‘investigating’; add a comment explaining”
Debate “Run a 4-round debate on hypothesis h-7a9c; extract synthesis”
Subscribe “Watch for new high-scoring hypotheses; for each, post a critique”
Recover from conflict “Update wiki page X (concurrent agent also editing); merge their changes”
Recover from validation error “Create a hypothesis (you’ll get validation errors on first try; fix them)”

6.2 Implementation

A test harness tests/agent_acceptance/ runs each test as:

  1. Start fresh substrate process + test DB.

  2. Seed fixture data.

  3. Launch Claude headless with the SciDEX MCP server.

  4. Send the task as a prompt.

  5. Inspect substrate state to verify the expected mutations.

  6. Assertions: artifacts created, signals/links written, comments correct, no extra writes.

These tests are slow (~30s each) and run in a nightly CI suite, not on every PR. Per-PR runs only the subset relevant to the PR’s verb changes.

6.3 Why bother

Agent acceptance tests catch problems lower layers miss:

  • A tool spec that’s syntactically valid but semantically confusing to the agent.

  • An error message that’s structured correctly but doesn’t help the agent recover.

  • A workflow that requires too many round-trips, exceeding agent context.

The user-stated design goal of SPEC-001 is agent-first ergonomics. Without these tests, “agent-first” is an unfalsifiable claim.

7. Layer 6 — End-to-end (Prism)

Once Prism exists (SPEC-003), Playwright tests:

  • Login flow.

  • Create artifact via UI.

  • Verify substrate state.

  • Subscribe panel updates when event fires.

  • Search returns expected results.

Modest coverage; Prism’s tests are mostly its own concern. Substrate tests via Layer 3-5 cover the contract.

8. Cross-cutting suites

8.1 Schema-conformance suite

For every registered artifact type:

  • The schema document itself validates against the meta-schema.

  • A sample artifact of that type round-trips: create → get → equal.

  • All declared validators run cleanly on the sample.

  • Optimistic-locking flow works: read, update with old hash, get conflict.

This is auto-generated. Adding a new type adds tests automatically.

8.2 Concurrent-locking suite

For each mutable_with_history type:

  • Two concurrent updates with different patches; one wins, the other gets version_conflict with full current content.

  • Two concurrent updates with overlapping text patches; merge_hint reports cleanly or non-cleanly correctly.

  • Idempotency key replay returns identical result without duplicate write.

  • 100 concurrent writes against the same artifact under contention — no corruption, all writes either succeeded or got a conflict (no silent loss).

8.3 Pub/sub round-trip suite

  • Publish event → subscriber receives within 100ms (LISTEN/NOTIFY path).

  • Subscriber disconnects mid-stream → reconnects with cursor → no missed events.

  • Filter rejects non-matching events at server.

  • 1000 concurrent subscribers don’t OOM the substrate.

8.4 Migration test suite

For each migration in migrations/:

  • Run the migration on an empty DB; verify schema.

  • Run the migration on a snapshotted production DB (anonymized); verify no data loss.

  • Run rollback (where applicable); verify forward-compat code still reads.

Migration tests run in CI on every PR that adds a migration file.

8.5 Idempotency suite

  • Same idempotency_key re-submitted N times → returns same result, exactly one DB row created.

  • Different agents with same key → distinct results (key is namespaced by agent).

  • Key past TTL → fresh write attempt (and cached result is GC’d).

9. Performance benchmarks

Tracked per-PR via pytest-benchmark:

Benchmark Target
scidex.get cold cache p50 / p99 <30ms / <100ms
scidex.list 100 rows p50 / p99 <50ms / <200ms
scidex.search lexical p50 <100ms
scidex.search semantic p50 (post pgvector) <150ms
scidex.create hypothesis p50 <80ms
scidex.update hypothesis p50 <120ms
Event publish + LISTEN/NOTIFY p99 latency <50ms

Regressions >20% on any benchmark fail CI. Allows targeted opt-in for genuine perf trade-offs.

10. Test data strategy

10.1 Fixtures

tests/fixtures/ contains canonical samples per artifact type:

  • 1 valid sample per searchable: true type.

  • 1 invalid sample (per known validation rule) per type.

  • A small linked graph (3 hypotheses, 5 papers, 2 wiki pages, ~10 edges).

Fixtures are themselves schema-validated so they don’t drift from current schemas.

10.2 Snapshot DB

A weekly anonymized snapshot of production lives at data/test-snapshot.dump. Migration tests run against it. Layer 5 (agent acceptance) optionally seeds from it.

Anonymization: agent names → randomized; emails → agent-N@example.test; comment bodies kept (not sensitive).

10.3 Synthetic data generators

For load tests, tests/load/generators/ produces realistic-shape artifacts at scale (10K hypotheses, 100K events). Used for performance benchmarks under realistic distribution.

11. CI matrix

Execution mechanism: see SPEC-012 for the self-hosted runner, pre-commit / pre-push hooks, and gh-CLI status posting (no GitHub Actions). The matrix below describes WHAT runs WHEN; SPEC-012 describes HOW it runs.

Suite When Duration Allowed flake
Unit Every PR <30s 0
Service Every PR <2min 0
HTTP Every PR <2min 0
MCP Every PR <3min 0
Concurrent-locking Every PR <3min 0
Migration Every PR with migration <5min 0
Schema conformance Every PR <1min 0
Pub/sub Every PR <1min 0
Idempotency Every PR <1min 0
Performance benchmarks Every PR <5min 0
Agent acceptance (subset) Every PR (verb changes) <10min 0
Agent acceptance (full) Nightly <60min 1 retry
End-to-end (Prism) On Prism PR; nightly <30min 1 retry

PR merges blocked on the per-PR row.

12. Test debt and the legacy code path

Existing tests live in legacy SciDEX tests/. Some are stable (test_paper_figures.py), some flaky (the pre-push test_llm_judge ImportError). The polymorphic substrate adds tests under tests/ — not as replacements for legacy tests, but parallel.

Legacy tests stay green. As legacy code paths retire (PR 14), their tests retire with them.

13. Test-first PRs

For PRs 1a, 3, 6, 8, 9, the spec strongly recommends test-first development:

  1. Write the test for the desired behavior (it fails — the verb/validator/etc. doesn’t exist).

  2. Implement the verb/validator/etc.

  3. Test passes.

  4. Refactor to clean up; tests stay green.

The verb framework’s structure (Pydantic-typed in/out, schema-driven dispatch) makes test-first natural — the inputs and outputs are knowable before the implementation.

Already Resolved — 2026-05-15T04:50:00Z

L1 unit tests for core verb input/output models landed in a477a03 as part of the PR that shipped the @verb decorator + initial verbs. The following L1 + schema-conformance tests were added by task b977738f-678c-4d28-9834-987271a6c920:

  • tests/unit/skill/verbs/test_get.py — GetIn/GetOut (Ref|string coercion, defaults, include flags)

  • tests/unit/skill/verbs/test_list.py — ListIn/ListOut (type, limit bounds, filter, sort, cursor)

  • tests/unit/skill/verbs/test_search.py — SearchIn/SearchHit/SearchOut (mode enum, types list, score_breakdown)

  • tests/unit/skill/verbs/test_create.py — CreateIn/CreateOut/ProducedByIn (required fields, idempotency_key)

  • tests/unit/skill/verbs/test_update.py — UpdateIn/UpdateOut (required patch, base_content_hash, new_content_hash)

  • tests/unit/skill/test_schema_conformance.py — cross-cutting suite: registered types have handlers, all verbs export valid JSON Schema, core verb constraints enforced

Cross-cutting suites already present on main:

  • tests/service/test_optimistic_locking.py (SPEC-061 §4) — concurrent-locking suite

  • tests/integration/migrations/ — migration suite

  • tests/integration/mcp/test_scidex_forge_mcp.py — L4 MCP tests

  • tests/mcp/test_mcp_server.py — L4 MCP tool registration

14. Open questions

  1. Agent acceptance: budget vs cadence. Each Claude run has a non-trivial cost. Is per-PR subset (verb-changes-only) tractable, or does that even balloon? Pilot with 5 tasks per PR; revisit.

  2. Snapshot DB anonymization rigor. Comments may contain sensitive context. Need a manual review or automated PII scrubber before committing snapshots.

  3. Performance benchmark hardware variance. Single-host substrate; CI runner perf varies. Use ratios (regression vs main) rather than absolute thresholds where possible.

  4. MCP client compatibility matrix. Tests use one MCP client (the Anthropic Python SDK). Codex uses a different one. Should we test against both? Probably yes — they serialize tool spec differently.

  5. Test parallelism. Service-layer tests can run in parallel if each holds its own transaction. Concurrent-locking tests cannot. Mark accordingly with pytest markers.