SPEC-012 — CI/CD (no GitHub Actions)

Self-hosted CI runner, local hooks, and the reproducible ci.sh pipeline.

Source: docs/design/spec-012-ci-cd.md

SPEC-012 — CI/CD Strategy (self-hosted)

Field Value
Status Draft v1
Owner kris.ganjam@gmail.com
Date 2026-04-29
Depends on SPEC-007
Pillar Cross-cutting

TL;DR

Zero-recurring-cost CI/CD for SciDEX and scidex-prism. Local hooks (pre-commit, pre-push) catch fast errors. A self-hosted runner on the SciDEX VPC executes the full test suite on PR pushes and posts status via gh CLI. No GitHub Actions; no cloud CI bills. Mirrors the existing fleet-health-watchdog pattern.

1. Goals

  • Zero recurring cost. No GH Actions, no third-party CI service.

  • Fast feedback. Format issues caught at commit; type/test issues at push.

  • Reliable. A green main is a buildable main.

  • Portable. The same checks run for any contributor; no “works on my machine.”

  • Visible. PR status updates without dashboard hunting — green check or red X on the PR like a normal CI.

2. Three layers

2.1 Local pre-commit (fast)

Runs in <2s. Format + lightweight lint. Refuses commits that fail.

2.2 Local pre-push (medium)

Runs in <30s. Type-check + unit tests on changed files. Refuses pushes that fail; override via SCIDEX_SKIP_PREPUSH=1 (consistent with existing ORCHESTRA_SKIP_PREPUSH=1).

2.3 Server-side runner (full)

Runs on every push to main and every PR commit. Full lint + type-check + tests + build. Posts status to PR via gh CLI. Lives on the SciDEX VPC.

3. Local pre-commit

Per-repo, in .githooks/pre-commit with git config core.hooksPath .githooks (so hooks are versioned with the repo).

3.1 SciDEX (Python)

#!/usr/bin/env bash
set -euo pipefail
ruff format --check .
ruff check .

3.2 Prism (TypeScript / Svelte)

#!/usr/bin/env bash
set -euo pipefail
pnpm prettier --check .
pnpm eslint --quiet .

Both fail-fast. Override: SCIDEX_SKIP_HOOKS=1 git commit ... (rare; user discretion).

4. Local pre-push

Per-repo, in .githooks/pre-push.

4.1 SciDEX

#!/usr/bin/env bash
set -euo pipefail
[ "${SCIDEX_SKIP_PREPUSH:-0}" = "1" ] && exit 0
ruff check .
mypy scidex/skill scidex/core
pytest tests/unit -x --timeout 30

4.2 Prism

#!/usr/bin/env bash
set -euo pipefail
[ "${SCIDEX_SKIP_PREPUSH:-0}" = "1" ] && exit 0
pnpm check
pnpm test --run

The override exists because pre-push isn’t always wanted (memory: existing scidex pre-push is broken in some env; ORCHESTRA_SKIP_PREPUSH=1 is the documented escape).

5. Server-side runner

A dedicated systemd service on the SciDEX VPC.

5.1 Architecture

┌──────────────────────────────────────────────────┐
│  scidex-ci.service (systemd timer, every 60s)     │
│  - gh pr list (open PRs in scidex, prism, ...)   │
│  - For each PR with new commits since last run:  │
│    1. Fresh worktree checkout                    │
│    2. Run repo-specific scripts/ci.sh            │
│    3. gh api repos/.../statuses/<sha> -f state=… │
│    4. Update local cache of "last tested SHA"    │
│  - Optional: comment with details on red runs    │
└──────────────────────────────────────────────────┘

5.2 Worktree pattern

Each PR run gets a fresh worktree:

git -C /var/lib/scidex-ci/scidex worktree add \
    /var/lib/scidex-ci/runs/scidex/PR-$NUM-$SHA $SHA
cd /var/lib/scidex-ci/runs/scidex/PR-$NUM-$SHA
./scripts/ci.sh
git -C /var/lib/scidex-ci/scidex worktree remove \
    /var/lib/scidex-ci/runs/scidex/PR-$NUM-$SHA

5.3 ci.sh per repo

SciDEX (scripts/ci.sh):

#!/usr/bin/env bash
set -euxo pipefail
ruff format --check .
ruff check .
mypy scidex
pytest tests/ -x --timeout 120 --cov=scidex --cov-report=term
python -c "import api"   # smoke import

Prism (scripts/ci.sh):

#!/usr/bin/env bash
set -euxo pipefail
pnpm install --frozen-lockfile
pnpm check
pnpm lint
pnpm test
pnpm build

5.4 Status posting

After ci.sh exits:

if [ $CI_STATUS = "success" ]; then
    gh api repos/$OWNER/$REPO/statuses/$SHA \
        -f state=success \
        -f context="scidex-ci/full" \
        -f target_url=$RUN_URL \
        -f description="All checks passed (${DURATION}s)"
else
    gh api repos/$OWNER/$REPO/statuses/$SHA \
        -f state=failure \
        -f context="scidex-ci/full" \
        -f description="$FAILURE_SUMMARY"
fi

Shows up as green check / red X on the PR.

5.5 Triggering

The runner polls every 60s. PR commenters wait <2min for status. Push-triggered immediate runs would require a webhook (external endpoint); polling is simpler and sufficient.

5.6 Concurrency

Default: one run at a time. Multiple PRs queue. With nproc≥8 hosts, 2-3 parallel runs are fine; queue bound prevents resource exhaustion.

6. Build & deploy

6.1 Build

After main is green, the runner produces deployable artifacts:

  • SciDEX: scidex Python wheel + restart-sensitive file diff log.

  • Prism: prism-build.tar.gz (the build/ directory after pnpm build).

6.2 Deploy

Production deploy is a separate concern; manual today, blue-green via systemd service swap in future. Out of scope for v1.

7. Per-repo conventions

7.1 SciDEX

scripts/ci.sh                # full test suite
scripts/install-hooks.sh     # `git config core.hooksPath .githooks`
.githooks/pre-commit         # ruff format + check
.githooks/pre-push           # ruff + mypy + pytest unit

7.2 Prism

scripts/ci.sh                # pnpm check/lint/test/build
scripts/install-hooks.sh     # `git config core.hooksPath .githooks`
.githooks/pre-commit         # prettier + eslint
.githooks/pre-push           # pnpm check + test

(Husky is an alternative for the JS-tooling crowd, but adds a dep. Plain .githooks + core.hooksPath is simpler and identical between repos.)

8. Failure modes

Failure Behavior
Runner host down Existing fleet watchdog detects via /health; restarts.
GitHub PAT expired Status post fails; runner logs error; on-call paged.
Test flakes Status flips green→red→green; PR author re-runs by pushing empty commit git commit --allow-empty -m 'rerun ci'.
Worktree disk fills systemd cleanup unit runs daily, removes worktrees older than 7 days.
Long-running test hangs per-test timeout in ci.sh kills hung suites.
Concurrent push to same PR runner detects new SHA, kills in-flight run for stale SHA, starts fresh.

9. Migration plan

# Title Scope Risk
C1 scripts/ci.sh in SciDEX Wraps existing checks. Run locally first to verify. Low
C2 scripts/ci.sh in Prism pnpm check/lint/test/build wrapper. Low
C3 .githooks/ for both repos pre-commit + pre-push. core.hooksPath set. Low
C4 scripts/install-hooks.sh One-line setup for new contributors. Low
C5 scidex-ci.service systemd unit Polls and tests; lives on the SciDEX VPC. Medium
C6 gh-CLI status posting First run posts to a test PR; verify status shows. Low
C7 Cleanup unit for worktrees systemd timer removing stale worktrees. Low
C8 Document hook installation in CONTRIBUTING.md One-line setup for new contributors. Low

Cadence: C1-C4 in week 1 (per repo). C5-C7 week 2. Stable by end of week 2.

10. Cost

  • Compute: existing SciDEX VPC. Runner uses idle cycles. Marginal cost: ~$0/month.

  • GitHub: gh-CLI uses standard PAT; no GH Actions billing.

  • Storage: ~10 GB for transient worktrees; cleaned daily.

Total: $0/month.

11. Open questions

  1. Self-hosted runner for forks: a PR from a fork can’t easily run the runner because the fork’s HEAD isn’t in the canonical clone. Workarounds: contributors push to a branch in the canonical repo (requires write access) or runner pulls from the fork (review carefully — running untrusted code).

  2. Concurrent runs: how many parallel test runs can the host handle? Probably 1-2; queue the rest.

  3. Test parallelism within a run: pytest -n auto from pytest-xdist for CPU-bound suites; I/O-bound suites don’t benefit as much. Evaluate per-repo.

  4. PR source for fork PRs: The runner can’t see commits from forks unless the fork is cloned. Fork contributors must push to a canonical branch or the runner must pull from the fork URL in the PR.

  5. Sticky status: once green, status is sticky. Only update on new commits. Avoids flapping when other PRs fail unrelated tests.

  6. Branch protection: main requires green status before merge on the GitHub side. This runner posts to the same status endpoint GH Actions uses — they stack cleanly.