Deploying Substrate

Production topology, db/role/env-file conventions, the scidex-substrate-v2.service unit, nginx wiring, /readyz vs /health.

Source: docs/tutorials/deploying-substrate.md

Deploying the SciDEX v2 Substrate

This tutorial covers the production deployment topology and the few concrete conventions a deployer needs to know to wire the substrate into a host that already runs services. It is intentionally short: exhaustive step-by-step procedures already exist in two places and should not be duplicated here.

For… Read…
Bare-metal host install end to end INSTALL.md
Single-host docker-compose stack docs/deploy/README.md
Recovering a broken deployment docs/runbooks/substrate_v2_rescue.md
The release-candidate gate docs/operations/v2-rc-checklist.md

This document is the deployer’s map: what the production topology looks like, what the conventions around databases and env files are, where the configuration files live in the repo, and which endpoint is the right one for each kind of health check.

1. Production topology

The v2 substrate runs as one FastAPI process under systemd, fronted by nginx, talking to a dedicated Postgres database. Per the deployment map in AGENTS.md:

                     ┌─────────────────────────────────┐
                     │  nginx (TLS, /api/*, /auth/*)    │
                     │  config: docs/deploy/nginx.conf  │
                     └────────────────┬────────────────┘
                                      │
            ┌─────────────────────────┼─────────────────────────┐
            ▼                         ▼                         ▼
   ┌────────────────┐       ┌─────────────────┐        ┌────────────────┐
   │ scidex-        │       │ persona-runner  │        │ ~60 scheduled  │
   │ substrate-v2   │       │ (pg_notify      │        │ worker units   │
   │ .service       │       │  consumer)      │        │ (debate orch,  │
   │ 127.0.0.1:8200 │       │                 │        │  gate evals,   │
   │                │       │                 │        │  ingestors)    │
   └────────┬───────┘       └────────┬────────┘        └────────┬───────┘
            │                        │                          │
            └─────────────┬──────────┴──────────────────────────┘
                          ▼
                ┌──────────────────────┐
                │ Postgres scidex_v2    │
                │ role: scidex_v2_app   │
                │ schema: public        │
                └──────────────────────┘

The only ingress is nginx → substrate (single uvicorn worker on 127.0.0.1:8200). Pub/sub is Postgres LISTEN/NOTIFY — no Redis or message broker. Rate limiting is an in-process token bucket — no sidecar. There is no required Forge service or Lambda. Workers run as their own systemd units and hit the substrate’s HTTP surface.

2. Database and role conventions

From AGENTS.md §Database:

Database Role Used by
scidex_v2 scidex_v2_app The production substrate. Live.
scidex_v2_dev any Local development.
scidex_v2_test (or any _test-suffixed name) any pytest. Required suffix; the guard in tests/conftest.py refuses non-_test databases.
scidex scidex_app v1 SciDEX production. Frozen, read-only reference. Never point v2 env vars at it.

The v1 read path used for ETL/migration goes through explicit env vars SCIDEX_V1_DSN / SCIDEX_V1_PG_RO_DSN, bound to a read-only role. The main SCIDEX_DSN must always point at v2 (or a _test database).

To create the database and role from a Postgres superuser shell:

CREATE ROLE scidex_v2_app WITH LOGIN PASSWORD 'change-me-in-prod';
CREATE DATABASE scidex_v2 OWNER scidex_v2_app;
ALTER  ROLE scidex_v2_app SET search_path = public;

The substrate creates its tables inside public — no extension schema required. No pgbouncer is required either; the substrate uses psycopg3’s async pool directly (see core/database.py). If you introduce pgbouncer, run it in transaction-pooling mode because the substrate’s per-request ContextVar discipline assumes one connection per request, not one per session.

3. Env-file conventions

Two conventions exist in the repo and they are not in conflict — they serve different consumers:

Path Read by Holds
/etc/scidex-substrate.env The main scidex-substrate-v2.service unit SCIDEX_DSN, SCIDEX_JWT_SECRET, OAuth creds, rate-limit knobs, the ~70 SCIDEX_* vars. Single file per INSTALL.md §4.
/etc/scidex-substrate/v2-db.env Companion service units that need only the DSN Just SCIDEX_DSN. Referenced from AGENTS.md.
/etc/scidex-substrate/cron.token Scheduled-worker .service units CRON_TOKEN — the bearer token for the system:cron actor. Per SPEC-043.
/etc/scidex-substrate/artifacts.env Workers that need S3 credentials AWS_*, SCIDEX_S3_*.
/etc/scidex-substrate/ingest.env Ingest workers Source-specific keys (PubMed, etc.). Per docs/operations/ingest-secrets.md.

All companion files live under /etc/scidex-substrate/ so that a single chgrp on the directory grants the substrate UID read access in one move. The main env file sits outside that directory because it predates the directory convention.

Every worker unit reads its env files with the - prefix so the unit doesn’t hard-fail on a missing file:

# from ops/systemd/scidex-ingest-pubmed-update.service
EnvironmentFile=-/etc/scidex-substrate/cron.token
EnvironmentFile=-/etc/scidex-substrate/artifacts.env
EnvironmentFile=-/etc/scidex-substrate/ingest.env

If a worker can’t get its bearer token, the request to the substrate falls back to anonymous and is rate-limited at the anonymous tier (SPEC-043 §1). This is intentional — workers degrade gracefully rather than crash-loop.

Set every env file to mode 0600 and owned by the substrate UID. The JWT secret is keying material for every authenticated request.

4. The main systemd unit

The reference unit is documented in INSTALL.md §7 Step 6:

# /etc/systemd/system/scidex-substrate-v2.service
[Unit]
Description=SciDEX Substrate v2
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=scidex
Group=scidex
WorkingDirectory=/opt/scidex/SciDEX-Substrate
EnvironmentFile=/etc/scidex-substrate.env
ExecStart=/opt/scidex/SciDEX-Substrate/.venv/bin/uvicorn \
    scidex_substrate.api.app:app \
    --host 127.0.0.1 --port 8200 \
    --workers 1 --proxy-headers
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

A few points worth knowing from the rest of the codebase:

  • One uvicorn worker. The substrate uses an in-process token-bucket rate limiter (no Redis), so multiple workers would each enforce a separate budget. If you need more throughput, scale horizontally with a shared rate-limit backend, not vertically with --workers N.

  • Restart=on-failure + SCIDEX_STARTUP_REQUIRE_MIGRATIONS=1. The combination means the unit crash-loops if migrations failed, rather than serving partial schema. The rescue runbook (linked below) covers the “silent death” case where systemd reports the unit active but uvicorn has wedged inside the lifespan handler.

  • Port 8200. The rescue runbook documents this as the canonical loopback port. nginx in docs/deploy/nginx.conf proxies to an upstream named substrate on port 8000 (the Dockerfile default); for a bare-metal deploy point the upstream at 127.0.0.1:8200 instead.

The companion ~60 worker units live under ops/systemd/ and are independent of the main API unit — install only the ones you actually need.

5. nginx

The canonical config lives at docs/deploy/nginx.conf. It documents and implements the route map:

Path Substrate endpoint Notes
/api/* FastAPI verb routes (one POST per @verb) SSE-aware; proxy_buffering off
/auth/* Auth router (dev login, OAuth, /auth/me)
/health Liveness — always 200 while the process is up
/health/deep Operator-facing deep status — 200 with degraded list on partial failure
/readyz Readiness — 503 with structured failure detail when migrations failed
/v1/readiness Public schema-drift detector
/v1/* Public REST surface (search, KG, wiki, papers, …)
/visual-artifacts/* Static images served by nginx from /data/scidex-visual-artifacts/ image MIME only

For TLS, drop fullchain.pem + privkey.pem into /etc/nginx/certs/ and uncomment the 443 block at the bottom of nginx.conf. Certbot can run alongside; mount its webroot if you want http-01 challenges to work.

client_max_body_size 10M is in the supplied config and is required for wiki bodies and evidence JSON. If you raise it, raise the substrate’s own request-validation limits too.

6. Health and readiness probes

The substrate exposes three health surfaces. They are deliberately distinct:

Endpoint Status code semantics Use for
GET /health Always 200 while the process is up. Reports version and verb_count. Process liveness. systemd.
GET /health/deep 200 always; reports a degraded: [...] list when a subsystem failed. Operator dashboards. Pages.
GET /readyz 200 ready / 503 with structured failure detail. Load-balancer health checks.

The LB rationale is documented at the top of docs/deploy/nginx.conf. /health stays 200 even when migrations failed at startup (so a partial-schema pod stays “alive” from systemd’s perspective and you can debug it); /readyz returns 503 in that case so the LB drains it. Wiring an LB at /health instead of /readyz is the failure mode that motivated incident #686 (M15-2). Use /readyz.

The deep-status report from /health/deep is also where operator runbooks land when they need to confirm “is this the build I think it is?” — it carries git_sha, last_migration, schema_applied / schema_expected, artifact_count, and agent_count. See _lifespan and the health_deep handler in api/app.py for the source.

For Prometheus, scrape GET /metrics — two series cover the HTTP surface (scidex_substrate_requests_total, scidex_substrate_request_duration_seconds). The full Grafana dashboard JSON is at docs/deploy/grafana/substrate_overview.json.

7. First boot: migrations and seeds

The FastAPI lifespan handler (_lifespan in api/app.py) runs three phases on every start, gated by env flags so production can opt out:

Flag Default Phase
SCIDEX_AUTO_MIGRATE 1 Apply pending migrations under src/scidex_substrate/migrations/ in numbered order. Tracks applied filenames in substrate_schema_migrations.
SCIDEX_AUTO_SEED_SCHEMAS 1 Seed default_schemas/*.json into the schema_registry table.
SCIDEX_AUTO_SEED_SKILLS 1 Seed skills/*/SKILL.md bundles into substrate_skills.
SCIDEX_AUTO_SEED_TOOLS 1 Seed tools/*/tool.yaml into tools.
SCIDEX_AUTO_SEED_QUALITY_GATES 1 Seed the 14 v1 quality gates (SPEC-024 §8).
SCIDEX_AUTO_SEED_V1_MISSIONS 1 Seed the 5 v1 missions + matching landscapes (SPEC-027).
SCIDEX_AUTO_SEED_PANTHEON_ARCHETYPES 1 Seed the curated pantheon archetypes (SPEC-028).
SCIDEX_AUTO_SEED_PERSONAS 1 Seed initial personas + wave-1 SPEC-020 personas.
SCIDEX_AUTO_SEED_LLM_PROVIDERS 1 Seed LLM providers + routing policies (SPEC-105).
SCIDEX_STARTUP_REQUIRE_MIGRATIONS 1 Refuse to boot if any migration failed. Set to 0 only for emergency boot — /readyz will report the partial state so the LB drains the pod.

Each seeder is idempotent (ON CONFLICT DO NOTHING or update-merge); re-running is cheap. Seeders that fail are logged but do not abort boot (a missing seed rarely makes the substrate unusable; a partial schema does).

For first-boot you usually want to apply migrations out-of-band so the report lands somewhere you can read:

cd /opt/scidex/SciDEX-Substrate
source .venv/bin/activate
scidex schema apply        # M26-5 unified CLI
# or:
scidex-substrate migrate   # legacy click CLI

Both are idempotent and exit zero.

8. Cutover from v1

The cutover plan lives in SPEC-006 and the day-of checklist is docs/operations/v2-launch-readiness.md.

Operationally:

  1. Stand up the v2 stack against scidex_v2 (this document).

  2. Run the verifier PYTHONPATH=src python3 scripts/ops/verify_substrate_ephemeral.py against an ephemeral DB; it should exit 0.

  3. Verify against the live scidex_v2: search, vote, fund, comment, /dashboard parity per docs/operations/v2-rc-checklist.md.

  4. Update DNS for scidex.ai → v2 LB.

  5. Leave v1 readable at v1.scidex.ai for archive access; mark its AGENTS.md and database read-only.

Per SPEC-015, v1’s data was backfilled into v2 in migration 0005. Substrate writes go to artifact_signals etc. — not the legacy v1 vote tables. There is no dual-write back to v1.

9. Repository pointers

Topic File
Full bare-metal install INSTALL.md
docker-compose deploy docs/deploy/README.md
nginx config docs/deploy/nginx.conf
Dockerfile Dockerfile
docker-compose docker-compose.yml
Reference env values .env.example
Production env-var inventory INSTALL.md §4
Companion systemd units ops/systemd/
Companion units rationale ops/systemd/README.md
Rescue runbook docs/runbooks/substrate_v2_rescue.md
Grafana dashboard docs/deploy/grafana/substrate_overview.json
API versioning policy docs/api/versioning_policy.md
RC gate docs/operations/v2-rc-checklist.md
Cutover plan SPEC-006
Operator CLI docs/operations/scidex-cli.md