SPEC-019 — Cutover Runbook

Historical cutover runbook preserved for invariants and rollback context; use current RC docs for live procedure.

Source: docs/design/spec-019-cutover-runbook.md

SPEC-019 — Cutover runbook: scidex.ai → v2

Field Value
Status Historical / superseded by current RC and cutover operations docs
Owner kris.ganjam@gmail.com
Date 2026-04-29
Depends on All v2 specs (SPEC-001 through SPEC-018)
Pillar Cross-cutting
Reverses All v1 entry points

Do not execute this file as the live cutover procedure. It records an early cutover model that assumed v2 would read the same Postgres database as v1. The current deployment runs v1 and v2 side by side with separate databases (scidex and scidex_v2). Use ../operations/v2-rc-checklist.md, ../operations/v2-launch-readiness.md, and ../cutover/v1-to-v2-cutover-plan.md for the live procedure.

TL;DR

Historical step-by-step procedure for an earlier plan to flip scidex.ai from v1 to v2. It is preserved for rationale and rollback thinking, not for direct operation.

0. Prerequisites

Before starting, confirm:

# substrate is healthy
curl -s https://substrate-staging.scidex.ai/health/deep | jq .

# expected: status='ok', last_migration='0012_*', verb_count >= 65,
# google_oauth_configured=false (still — fix in step 2)

# economic-loop smoke test passes against the SHARED Postgres
SCIDEX_TEST_DSN='postgresql://...prod...' \
  pytest tests/service/test_economic_loop_smoke.py -x

# expected: 3 passed

If any of these fail, DO NOT proceed. Fix root cause first.

1. Snapshot the world

# Tag the current v1 deploy commits
git -C /opt/scidex-v1 tag pre-cutover-$(date +%Y%m%d)
git -C /opt/scidex-substrate tag cutover-substrate-v1.0
git -C /opt/scidex-prism    tag cutover-prism-v1.0

# Postgres backup (already happens daily via configs/backups/scidex.yaml,
# but force a fresh one immediately before cutover)
sudo -u postgres pg_dump -Fc scidex > \
  /data/backups/scidex-pre-cutover-$(date +%Y%m%d-%H%M).dump

Cutover preserves all data — v2 writes to the same scidex database v1 reads from. The backup is insurance only.

2. Configure v2 secrets

On the substrate host:

# Real Google OAuth (PR 73)
export GOOGLE_OAUTH_CLIENT_ID=...
export GOOGLE_OAUTH_CLIENT_SECRET=...
export GOOGLE_OAUTH_REDIRECT_URI=https://scidex.ai/auth/oauth/google/callback

# JWT signing — REUSE v1's secret if any so existing tokens remain valid.
# If v1 didn't issue JWTs, generate a fresh one:
export SCIDEX_JWT_SECRET="$(openssl rand -hex 32)"

# Agent runtimes own provider credentials and persona judgment loops.
# Configure those in scidex-agents, not in the substrate service env.

# DB pointer — same as v1
export SCIDEX_DSN='postgresql://...'

# Feature flags (defaults are sensible; adjust if needed)
# export SCIDEX_SUBSTRATE_AUTO_PROMOTE=1   # default
# export SCIDEX_SUBSTRATE_AUTO_PAYOUT=1    # default
# export SCIDEX_SUBSTRATE_AUTO_CALIBRATE=1 # default
# export SCIDEX_SUBSTRATE_AUTO_REFUND=1    # default
# export SCIDEX_DEV_AUTH=0                 # turn OFF in production

Restart substrate:

systemctl restart scidex-substrate

Verify:

curl -s https://substrate.scidex.ai/health/deep | jq .
# google_oauth_configured: true  ← MUST be true now
# persona_runner.runtime_owner: scidex-agents
# system_policies.system_policies: [configured entries]

3. Wire up cron jobs

Two periodic workers ship in v2; operators run them on cron:

# Score-history snapshots (PR 71). Cadence: every 30 minutes.
*/30 * * * * scidex-app cd /opt/scidex-substrate && \
  python -c 'from scidex_substrate.score_history import snapshot_active_scores; \
             import asyncio, psycopg, os; \
             asyncio.run((async def(): \
               c=await psycopg.AsyncConnection.connect(os.environ["SCIDEX_DSN"]); \
               print(await snapshot_active_scores(c)); \
               await c.close())())'

# Senate sweep (PR 75). Cadence: hourly.
0 * * * * scidex-app cd /opt/scidex-substrate && \
  python -c 'from scidex_substrate.senate_sweep import sweep_overdue_proposals; ...'

(Real cron lines are longer — see docs/deploy/cron.md for templates.)

4. Smoke test against production

# End-to-end happy path against the real DB
SCIDEX_TEST_DSN="$SCIDEX_DSN" \
  pytest tests/service/test_economic_loop_smoke.py -x -v

# expected: 3 passed (test creates + cleans up its own data)

# Manual UI check via the Prism build
PRISM_PROXY_BASE=https://substrate.scidex.ai \
  npm --prefix /opt/scidex-prism run build && \
  systemctl restart scidex-prism

# Visit https://prism-staging.scidex.ai (DNS preview before public flip)
# Verify:
#   - /  loads dashboard with stats + competitive surfaces
#   - /login shows Google sign-in button
#   - Sign in via Google
#   - /personas shows 6 personas with decision skills
#   - /leaderboard shows multiple boards including funder + skill
#   - /trending shows movement
#   - Visit a knowledge_gap → fund → see auto-promote → submit answer →
#     resolve → confirm dividend signal on winner

5. DNS flip

# Lower TTL on scidex.ai 24h before cutover so the swap propagates fast
# (do this before step 0 if possible)

# Switch the A/CNAME records
#   scidex.ai      → v2 reverse proxy (was → v1)
#   v1.scidex.ai   → v1 reverse proxy (new alias)

# Cloudflare / Route 53 / etc: update via console or CLI.
# Example with Cloudflare CLI:
flarectl dns u --domain scidex.ai --name scidex.ai --type A --content $V2_IP
flarectl dns u --domain scidex.ai --name v1 --type A --content $V1_IP

DNS propagation: usually <60s with low TTL; full settle <5 min.

6. Reverse-proxy update

If you’re using a single reverse proxy (nginx / Caddy / Traefik):

# /etc/nginx/sites-enabled/scidex.ai
# scidex.ai now proxies to Prism (v2 frontend);
# Prism's handleFetch proxies /api/scidex/* + /auth/* to substrate.

server {
  server_name scidex.ai;
  listen 443 ssl http2;

  # ... TLS config ...

  location / {
    proxy_pass http://prism-v2-internal;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

server {
  server_name v1.scidex.ai;
  listen 443 ssl http2;
  # ... TLS config ...
  location / {
    proxy_pass http://v1-internal;
  }
}

Reload:

nginx -t && systemctl reload nginx

7. Watch the rollout

For ~1h after flip:

# Agent runtime activity
cd /home/ubuntu/scidex-agents
./runtimectl status
./runtimectl logs agent-jerome

# Substrate access logs
journalctl -u scidex-substrate -f | grep -E '(ERROR|WARN)'

# Postgres slow queries
sudo -u postgres psql -c "SELECT * FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10"

# /health/deep degradation watch
watch -n 30 'curl -s https://scidex.ai/health/deep | jq .status'

Indicators of trouble:

  • 5xx rate >0.5% — substrate or DB issue; rollback (see §8)

  • Agent runtime crashloop — check ./runtimectl status, runtime logs, and provider credentials in the agent repo

  • OAuth redirect_uri mismatch — operator typo in env var; fix and restart

  • Slow queries on artifact_signals — score-cache trigger lag; manually REFRESH MATERIALIZED VIEW if needed

8. Rollback procedure

If anything goes wrong:

# 1. Flip DNS back
flarectl dns u --domain scidex.ai --name scidex.ai --type A --content $V1_IP

# 2. Stop the v2 substrate (optional but cleaner)
systemctl stop scidex-substrate

# 3. Verify v1 is serving
curl -s https://scidex.ai/api/v1/health

Data is safe — both v1 and v2 wrote to the same Postgres. v2’s substrate-side tables (substrate_actor_links, substrate_provenance_links, substrate_artifact_links, substrate_score_history, substrate_notification_cursors) are append-only and don’t conflict with v1.

9. Post-cutover cleanup (week 2+)

Once v2 has been live and stable for a week:

  • Lower v1.scidex.ai’s traffic priority; communicate that read-only archive remains accessible

  • Notify users via the existing comment/notification infrastructure

  • Disable v1’s write paths (remove its API endpoints from nginx)

  • Eventually decommission v1 hardware

10. Open issues to track

  • Wiki page rendering parity — verify mermaid + PMID render correctly on /wiki_page/. If issues found, the v1 fix from PR 1018 may need to be ported.

  • Forge tools migration — 58 v1 tools to register as v2 tool artifacts. Mechanical; can be done post-cutover.

  • Debate session orchestration — multi-persona structured debates. Future PR.

11. PRs that built this state

Pillar Substrate PRs Prism PRs
Polymorphic substrate 1-15 1-10
Provenance pipeline 48-53 39-40
Persona LLM-decide 54-56 41-43
Closed economic loop 57-69 44-47
Competitive surfaces 66-68, 79 48-49, 53
Real OAuth 73 51
Senate workflow 74-75, 77 52
Smoke + verification 76 —
Adoption + ops 70-72, 78, 80 50

(Numbers approximate; see git log for canonical PR list.)