v1 → v2 Cutover Plan

Phase A/B/C plan and nginx flip ordering for the public substrate cutover.

Source: docs/cutover/v1-to-v2-cutover-plan.md

v1 -> v2 Cutover Plan

Status (2026-05-16): RC §9 production-prerequisite gates are all closed. The public scidex.ai flip is now gated by three process validations: live restore drill, Prism-side DID login form, and live peer federation round-trip.

The user’s directive (“when you are done, you can re-route scidex.ai to the new site and rename the old site to v1.scidex.ai”) remains the terminal cutover action. This runbook describes the current nginx-backed procedure.

1. Readiness State

The authoritative gate table is docs/operations/v2-rc-checklist.md. As of the post-#1374 audit:

Gate area State Evidence
RC §1-§8 Closed M1-M7 spec/code/integration surfaces are marked [x]; smoke evidence is PR #1351 8dc5f876 with 99.0% pytest pass.
RC §9.1 UI auth substrate support Closed JWT cookie/bearer login/refresh/logout/revocation landed; lifecycle tests landed.
RC §9.2 Federation peer-sync code Closed Outbound publisher, scheduled worker, inbound pull, peer registry verb, runbooks, and commit-log backup landed.
RC §9.3 x402 live Closed Live receipt/dispute endpoints write receipts and ledger rows.
RC §9.4 Public read API Closed Public middleware, Redis-backed per-IP rate limits, 429 tests, and OpenAPI route are on main.
RC §9.5 Monitoring + alerting Closed Alert surfaces, /healthz, /metrics, on-call docs, and runbook index are on main.
RC §9.6 Backup strategy Closed scripts/ops/backup.sh and docs/runbooks/emergency_restore.md landed.
RC §9.7 Runbooks Closed Required runbooks, on-call handover, escalation chain, and post-mortem scaffold landed.

Remaining Process Gaps

These are not new substrate-code gates, but they must be complete before the operator calls the flip:

  1. Live restore drill: execute docs/runbooks/emergency_restore.md against a real backup on a non-production PG instance, and record the backup path, target DSN, operator, and verification result.

  2. Prism-side DID login form: verify the Prism UI flow where an unauthenticated visitor reaches /login, completes DID-backed login, and lands on /feed with a working signed session. No v1 cookie fallback.

  3. Live peer federation round-trip: register a real peer, publish a peer edit, ingest it through v2 federation, and confirm it appears in prism.scidex.ai within the configured pull interval.

2. Current v2 Surface

Core Primitives

Primitive Substrate verbs Prism surface Notes
Wikis yes /wiki, /wiki/[slug] Markdown render, search, entity-type filter.
Hypotheses yes /hypothesis, /hypothesis/[id] Score grid, evidence, market badge.
Papers yes /papers, /papers/[id] PubMed/DOI/PMC links.
Knowledge gaps yes /gap, /gap/[id] Create-bounty workflow.
Colliders yes /collider, /collider/[id] Bet and resolve lifecycle.
Bounty challenges yes /challenges, /bounty_challenge/[id] Submit, judge, award, dividend.
Missions yes /mission, /mission/[id] Fund, activate, complete.
Pantheon debates yes /pantheon, /pantheon_debate/[id] Open, argue, signal, resolve.
Arenas yes /arena, /arena/[id] Pairing, Elo, judge.
Senate yes /senate Proposals, gates, ops dashboards.
Benchmarks yes /benchmark, /benchmark/[id] Submit and score.
Squads yes /squads, /squad/[id] Findings and journal.

Dashboards And Cross-Primitive Views

  • /economy — token economy dashboard, dividends, top earners.

  • /leaderboard — funded gaps, hypotheses, challenges, wikis, replicated hypotheses, funders, forecasters, skills, producers, replicators.

  • /knowledge_graph — graph viewer.

  • /contested — contested artifact pulse.

  • /dashboard, /trending, /feed — entry surfaces.

  • /actor/[id] — actor profile and dividend tab.

  • /forge, /forge/landscape, /forge/playground — tool discovery and use.

Operational Surfaces

  • GET /health — liveness.

  • GET /readyz — readiness; load balancers should use this, not /health.

  • GET /healthz — DB liveness.

  • GET /metrics — Prometheus metrics.

  • POST /api/scidex/* — polymorphic verbs through Prism nginx.

  • POST /auth/* — substrate-backed auth.

3. Ready-To-Flip Checklist

Ops must complete exactly these three actions before running the cutover:

  1. Run the live restore drill from docs/runbooks/emergency_restore.md against a real backup and record the result in the release notes.

  2. Pass the Prism DID login smoke from a clean browser session: https://prism.scidex.ai/login -> DID-backed login -> /feed -> one authenticated session refresh or write-path API check.

  3. Pass the live peer federation round-trip using docs/operations/atproto-peer-registry-runbook.md: peer write -> inbound pull -> visible artifact on prism.scidex.ai.

4. Cutover Procedure

The current flip is an nginx vhost swap on the existing host. DNS for v1.scidex.ai is prepared first; scidex.ai itself keeps pointing at the same host/load balancer. The script then makes scidex.ai serve v2 and moves the legacy v1 app to v1.scidex.ai.

Pre-Cutover

  1. Confirm the three ready-to-flip checklist items above are done.

  2. Add DNS A record:

    v1.scidex.ai -> <current scidex.ai host public IP>

  3. Expand the cert SAN set:

    sudo certbot certonly --expand \
  -d scidex.ai -d www.scidex.ai \
  -d v1.scidex.ai \
  -d prism.scidex.ai \
  -d asta.scidex.ai \
  -d orchestra.scidex.ai

  4. Verify DNS, cert, and v2 health:

    dig +short v1.scidex.ai

openssl x509 -in /etc/letsencrypt/live/scidex.ai/fullchain.pem -noout -text \
  | grep -q 'DNS:v1.scidex.ai'

for p in / /wiki /hypothesis /papers /gap /economy /senate /collider \
         /challenges /pantheon /arena /trending; do
  curl -sS -o /dev/null -w "$p:%{http_code}\n" "https://prism.scidex.ai$p"
done

  5. Take a pre-cutover database snapshot:

    pg_dump scidex_v2 -F c \
  -f /data/backups/sandbox/scidex_v2_pre_cutover_$(date +%Y%m%d_%H%M).dump

Flip

Run:

sudo bash docs/cutover/staged/flip.sh

The script performs the actual switch:

  • backs up /etc/nginx/conf.d/scidex.conf under /etc/nginx/conf.d/.cutover-backup/;

  • installs docs/cutover/staged/v1.scidex.ai.conf, routing v1.scidex.ai to the legacy v1 FastAPI service on 127.0.0.1:8000;

  • installs docs/cutover/staged/scidex.ai.v2.conf, routing scidex.ai /api/* and /auth/* to substrate on 127.0.0.1:8200, and all other app traffic to Prism on 127.0.0.1:3200;

  • updates scidex-prism-v2.service so ORIGIN=https://scidex.ai;

  • runs nginx -t, reloads nginx, restarts Prism, and smokes both apex v2 and v1.scidex.ai.

Post-Cutover Watch

For the first hour:

journalctl -u scidex-prism-v2 -u scidex-substrate-v2 --since '1 hour ago' \
  | grep -iE 'error|exception'

Smoke the public paths:

for path in / /wiki /hypothesis /papers /gap /economy /senate /collider \
            /challenges /pantheon /arena /feed; do
  code=$(curl -sS -o /dev/null -w "%{http_code}" -L "https://scidex.ai$path")
  echo "$path -> $code"
done

Confirm v1 remains reachable:

for path in / /vision /dashboard /agora; do
  code=$(curl -sS -o /dev/null -w "%{http_code}" -L "https://v1.scidex.ai$path")
  echo "$path -> $code"
done

5. Rollback

Rollback window: keep the v1 service, cert SANs, and nginx backup intact for at least the first 24 hours after the flip. Normal rollback is expected to be a quick nginx/service-origin revert during that watch window.

Rollback owner: T1 substrate on-call engineer. Escalate immediately to T2 substrate lead for any P0, failed rollback command, suspected data-integrity issue, or unresolved user-facing outage. The current rotation and escalation chain are in docs/operations/oncall-handover.md.

Rollback command:

sudo bash docs/cutover/staged/flip.sh --rollback

This restores the saved v1 scidex.conf, removes the v1.scidex.ai vhost, resets Prism ORIGIN to https://prism.scidex.ai, restarts Prism, and reloads nginx.

Database restore is not part of routine web rollback. Use docs/runbooks/emergency_restore.md only for confirmed data corruption or an explicit operator decision.

6. Deferred Work After Flip

These do not block the public apex move:

  • Raw Forge tool count can continue growing toward v1’s long tail.

  • Static/minor v1 pages (/quests, /docs, /specs, /contributors, /atlas, /datasets, /clinical-trials, /protein-designs) can continue shipping as Prism routes where product value justifies them.

  • Older cosmetic nav and homepage polish remains non-blocking if current smoke surfaces are healthy.

7. Decision Rule

Call the flip only when:

  • all three ready-to-flip checklist actions are recorded;

  • prism.scidex.ai smoke routes return expected 200s;

  • v1.scidex.ai DNS and cert SAN are live;

  • the pre-cutover scidex_v2 snapshot exists;

  • T1 substrate on-call is present for the first-hour watch and can execute rollback if needed.

At that point the remaining risk is operational, bounded by the 24-hour rollback window and the staged flip.sh --rollback path.