Runbook — Artifact Recovery

Recover missing or suspect artifact payloads into approved v2 stores without mutating v1.

Source: docs/runbooks/artifact-recovery.md

Artifact Recovery Runbook

Use this runbook when a v2 artifact row exists but its payload, manifest, repo link, or blob registration is missing or suspect.

The recovery rule is simple: preserve provenance, restore bytes into an approved v2 store, and register the result in Substrate. Do not repair v2 by writing new bytes into v1 artifact paths or archived GitHub repositories.

Approved Recovery Targets

Payload shape Target
Large binary, model weights, PDFs, raw datasets S3 through substrate blob-store helpers.
Sparse stable bundle, recovered legacy payload, small notebook/figure bundle Forgejo aggregate repo https://forge.scidex.ai/scidex-artifacts/scidex-artifacts.git.
Living paper, wiki, hypothesis, review, mission, or collaborative analysis Artifact-specific Forgejo repo.

Never write recovered v2 payloads to:

  • /data/scidex-artifacts;

  • /data/artifacts;

  • /home/ubuntu/scidex/data/scidex-artifacts;

  • archived GitHub artifact repositories.

Triage Checklist

  • Confirm the artifact ref, type, current version, lifecycle state, and expected payload fields.

  • Confirm whether the artifact is active, superseded, archived, quarantined, or already replaced.

  • Find the expected payload location from Substrate metadata, Forgejo repository records, S3 blob records, or bundle manifests.

  • Check whether the missing bytes exist in an approved v2 store before consulting v1.

  • If v1 is needed, use it as read-only provenance. Do not mutate v1.

  • Record the original path, source commit or database row, checksum, and why recovery is needed.

Recovery Procedure

  1. Stop writers if the artifact is actively changing. For high-value artifacts, pause the relevant worker, mission, or agent packet before replacing payload pointers.

  2. Copy bytes into an approved v2 store. Use S3 helpers for blobs or a Forgejo worktree for bundle content. Keep the original file untouched.

  3. Compute checksums and size. Record SHA-256 or the blob-store checksum returned by the helper. Do not infer file integrity from content_hash.

  4. Create or update a manifest. The manifest must include:

    • artifact ref and version;

    • source location and source checksum when available;

    • target bucket/key/version or repo/path/commit;

    • recovered size and checksum;

    • recovery reason;

    • producer/operator identity;

    • timestamp.

  5. Register the recovered location in Substrate. Update only the v2 artifact metadata or location artifact needed to point at the recovered payload.

  6. Verify readback. Fetch the artifact through the public API/verb path, not only by opening files on disk.

  7. Leave an audit note. Link the recovery manifest from comments, artifact history, or an operations report so future reviewers know why the location changed.

content_hash Warning

The v1 docs captured an important incident: content_hash was treated as a file checksum and rewritten during repair. That was wrong. In both v1 and v2, content_hash is a semantic dedupe or optimistic-lock key. File integrity must use explicit blob checksum fields such as metadata.file_sha256, S3 checksum, or a manifest checksum.

Do not write a recovery tool that recomputes content_hash from file bytes.

v1 Recovery Source Rules

If the only copy is in v1:

  • connect to v1 only through an explicitly configured read-only DSN;

  • copy the payload into an approved v2 store;

  • preserve the original v1 path and row/ref in the recovery manifest;

  • do not commit to v1 artifact repos;

  • do not update v1 database rows;

  • do not claim v1 payload paths are durable v2 locations.

Validation

At minimum:

PYTHONPATH=$PWD/src pytest tests/unit

For recovery code or migration helpers, also run the targeted integration or service test with a _test database. Never validate by pointing pytest at scidex or scidex_v2.

Related