Runbook — Federation Live Verification

End-to-end live test of cross-peer federation handshake and replication.

Source: docs/runbooks/federation-live-verification.md

Federation Live Verification Runbook

Purpose: Verify that AT Protocol federation (outbound publishing + inbound pull) works in a live or near-live environment.

Trigger: Parity Blocker 5 verification, RC §9.2 process gap closure, or any time you suspect federation is broken.

Prerequisites

  • Access to the substrate deployment (local dev or staging)

  • SCIDEX_DSN pointing to a live Postgres instance with migrations applied

  • PYTHONPATH=src set for all pytest invocations

Outbound Publisher — Smoke Test

The outbound publisher consumes events table rows with federation_level='public', builds signed AT Proto commits, and appends them to at_proto_commits.

Step 1: Publish a public artifact event

# Using the scidex skill verbs (if API is up) or direct SQL:

# Option A — via HTTP API (if substrate is running):
curl -s -X POST http://localhost:8000/v1/agents/call \
  -H "Content-Type: application/json" \
  -d '{
    "verb": "scidex.create",
    "input": {
      "type": "hypothesis",
      "federation_level": "public",
      "title": "TREM2 R47H impairs microglial activation",
      "createdAt": "2026-05-13T12:00:00+00:00"
    }
  }' | jq .

Step 2: Trigger the federation commit publisher

# The runner exposes a run_once() entry point.
# In production this runs as a scheduled worker; for testing invoke directly:

PYTHONPATH=src python -c "
import asyncio
from scidex_substrate.scheduled_workers.federation_commit_publisher import policy
from scidex_substrate.scheduled_workers.federation_commit_publisher.runner import run_once

async def trigger():
    import psycopg
    SCIDEX_DSN = 'postgresql:///scidex_v2'  # or \$SCIDEX_DSN
    async with await psycopg.AsyncConnection.connect(SCIDEX_DSN) as conn:
        pol = policy.policy_from_env()
        result = await run_once(conn, pol=pol, dry_run=False)
        print(f'Published: {result.items_produced}, status: {result.status}')

asyncio.run(trigger())
"

Step 3: Verify a row landed in at_proto_commits

psql "$SCIDEX_DSN" -c "
SELECT commit_cid, repo_did, rev, collection, rkey, source_peer_did, ts
FROM at_proto_commits
WHERE ts > NOW() - INTERVAL '5 minutes'
ORDER BY ts DESC
LIMIT 10;
"

# Expected: >= 1 row with collection = 'app.scidex.hypothesis' (or whatever artifact type you published)

Step 4: Verify federation.commit_published event fired

psql "$SCIDEX_DSN" -c "
SELECT id, event_type, source, payload
FROM events
WHERE event_type = 'federation.commit_published'
  AND created_at > NOW() - INTERVAL '5 minutes'
ORDER BY created_at DESC
LIMIT 5;
"

# Expected: 1 row; payload.commit_cid matches a commit_cid from Step 3

Inbound Subscriber — Unit Test (Mocked Firehose)

The inbound subscriber ingests commits from remote peers via the ATProtoSyncCommitsService. Live-fire testing requires a real AT Proto network peer, so this runbook uses the unit test suite with a mocked firehose source.

Run the inbound unit tests

PYTHONPATH=src pytest tests/unit/application/federation/test_sync_commits.py -v

Expected: 38 passed.

What the tests cover

Test What it verifies
test_valid_commit_is_ingested_and_event_emitted A valid AT Proto commit writes to at_proto_commits and emits a *.federated_in event
test_resume_after_restart_resumes_from_last_rev Cursor advances correctly on restart
test_blocked_peer_rejects_all_commits Blocked peer commits are drained and rejected
test_stale_rev_is_rejected Monotonic rev gate works
test_mixed_batch_partial_ingestion Partial batches are handled correctly
test_counter_increments_persist_across_two_syncs Ingest/reject counters accumulate
test_multi_op_commit_emits_one_event_per_op Multi-op commits fan out correctly

Manual inbound test with a fake commit

# Minimal script to inject a fake commit and verify it lands
import asyncio
from datetime import UTC, datetime
from scidex_substrate.application.federation.sync_commits import ATProtoSyncCommitsService
from scidex_substrate.federation.types import AtProtoCommit, AtProtoOperation

# Fake firehose source
class FakeSource:
    async def subscribe(self, peer_did, peer_url, *, cursor, limit):
        yield AtProtoCommit(
            cid="bafyreitest123",
            did=peer_did,
            rev="00000000000000000001",
            ts=datetime(2026, 5, 13, tzinfo=UTC),
            ops=[
                AtProtoOperation(
                    action="create",
                    collection="app.scidex.hypothesis",
                    rkey="h-test-1",
                    cid="bafyreirecord001",
                )
            ],
            prev_commit_cid=None,
        )

# Fake DB connection (in-memory fake from tests/unit/application/federation/test_sync_commits.py)
# For manual verification, use the unit test framework rather than scripting

# Expected result after running: 1 row in at_proto_commits, 1 hypothesis.federated_in event

Integration Smoke Test (requires live Postgres)

# Requires SCIDEX_TEST_ADMIN_DSN set
SCIDEX_TEST_ADMIN_DSN=postgresql:///postgres \
PYTHONPATH=src pytest tests/integration/federation/test_atproto_publish_smoke.py -v

Expected: All 4 scenarios pass.

Scenarios covered:

  1. Public artifact triggers publish end-to-end

  2. Private artifact is not published

  3. Rev is monotonic across multiple events

  4. Unmapped event type skips without blocking

Parity Inventory Status

After verifying, update docs/operations/v1-parity-gaps.md line 355:

| Federation peer sync | DONE | Full outbound + inbound pipeline confirmed via unit tests; integration smoke requires live PG |

Troubleshooting

at_proto_commits shows 0 rows after publishing

  1. Check that the artifact event has federation_level='public':

    SELECT id, event_type, payload->>'federation_level' AS fed_level
FROM events WHERE payload->>'id' = '<artifact_id>';

  2. Verify the publisher didn’t hit the kill switch:

    echo $SCIDEX_SCHED_FEDERATION_COMMIT_PUBLISHER_ENABLED  # must not be 'false'

  3. Check the events table for the artifact — if federation_level is absent or 'private', it won’t publish.

Inbound test failures

  1. test_unknown_peer_raises — ensure the peer is registered in federation_peers:

    SELECT * FROM federation_peers WHERE peer_did = 'did:plc:...';

  2. test_stale_rev_is_rejected — check federation_peer_state.last_rev for the peer:

    SELECT last_rev FROM federation_peer_state WHERE peer_did = 'did:plc:...';

  3. Lexicon validation failures — verify lexicons are loaded:

    from scidex_substrate.federation.lexicons import load_all
lexicons = load_all()
print(list(lexicons.keys()))  # should include app.scidex.hypothesis etc.

Pre-existing test failures (not federation-related)

test_translator.py::test_translate_matches_golden[*] failures are schema drift in the golden files — unrelated to federation. These are tracked separately and do not block the federation verification.

Related Runbooks