Artifact Location Migration

SciDEX artifact identity stays in Substrate. Git and S3 locations are mutable storage pointers recorded in artifact metadata and in each sparse bundle's

Source: docs/artifact-location-migration.md

Artifact Location Migration

SciDEX artifact identity stays in Substrate. Git and S3 locations are mutable storage pointers recorded in artifact metadata and in each sparse bundle’s artifact_locations.json.

Scope

This migration path is for existing sparse artifact bundles, especially the GitHub/S3 scidex-artifacts collection. It is not the default authoring flow for new long-lived scientific documents. New living artifacts should eventually ask Substrate to create or bind a Forgejo working copy, then update through branches, pull requests, reviews, and webhook ingest.

The sparse migration tool keeps refs stable:

  • Substrate keeps the canonical artifact ref and permissions.

  • GitHub remains a source/fallback location for scidex-artifacts.

  • S3 remains the blob/payload store where applicable.

  • Forgejo becomes the active Git location after push and sparse checkout verification.

Repository Policy

  • Use collection repositories for singleton/simple artifacts and sparse bundles: wiki pages, hypotheses, protein designs, notebooks, analyses, work packets, evidence links, and other artifacts that usually do not need an independent issue/PR/release lifecycle.

  • Use per-artifact repositories only when the artifact is a living project: papers, large analysis bundles, datasets, and artifacts that need their own branches, reviews, releases, or collaborators.

  • The default Forgejo collection target is https://forge.scidex.ai/scidex-mirrors/scidex-artifacts.git.

  • The default source locations remain https://github.com/SciDEX-AI/scidex-artifacts.git and s3://scidex-artifacts/<bundle-path>/ until the Forgejo push and sparse checkout both verify.

Location Record

artifact_locations.json uses schema=scidex-artifact-locations@v1.

Important fields:

  • primary_ref: the main artifact ref for the bundle.

  • artifact_refs: every SciDEX ref represented by the bundle.

  • active_git_location: the active Git provider name, usually github before migration and forgejo after verification.

  • locations[]: provider records with repository URL, commit, bundle path, sparse checkout patterns, state, and provider-specific owner/repo metadata.

  • blob_locations[]: S3 or other blob-store pointers that travel with the bundle.

When a location record is embedded inside the Git commit that contains the record, that location may use:

{"commit_sha": "HEAD", "commit_sha_kind": "enclosing_ref"}

Substrate metadata should receive exact commit hashes. Embedded HEAD avoids a self-referential commit problem inside the repository copy itself.

artifact_location_active is the current structured pointer. Older sparse bundles may also carry sparse_artifact_repo; treat it as a compatibility alias until all readers are migrated.

Tooling

Use:

./runtimectl migrate artifact-location \
  --primary-ref TYPE:ID \
  --bundle-path TYPE/ID \
  --ref OTHER:ID \
  --repo-mode auto \
  --ensure-forgejo-repo \
  --push \
  --update-scidex

--repo-mode auto chooses collection repositories for simple artifacts and per-artifact repositories for papers, datasets, and analysis bundles. Override with --repo-mode collection or --repo-mode per-artifact when needed.

The tool writes:

  • artifact_locations.json inside the source sparse bundle.

  • artifact_locations metadata on updateable SciDEX artifacts.

  • A Forgejo bundle copy under the collection repo path matching the source sparse path, or under artifact/ for per-artifact repositories.

  • Prism review URLs for every supplied artifact ref.

Operational Rules

  • Start with a clean source sparse repo or explicitly inspect unrelated local changes before running with --commit-source-manifest.

  • Use --dry-run first for new artifact classes or org mappings.

  • Use --agent-ref to record the responsible agent in Forgejo commit trailers. The default is a generic migration operator; Kris-specific runs should pass --agent-ref agent:persona-kris-ganjam.

  • --update-scidex requires a SciDEX API key. The generic default env var is SCIDEX_AGENT_API_KEY; pass --api-key-env SCIDEX_AGENT_API_KEY_KRIS for Kris-owned repairs.

  • Forgejo writes require SCIDEX_FORGEJO_TOKEN_FILE or --forgejo-token-file; raw token values must not be logged or committed.

  • If Forgejo push succeeds but SciDEX metadata update fails, the location record is still recoverable from the source bundle and Forgejo bundle. Re-run with --update-scidex after fixing credentials or content hashes.

  • If active Forgejo verification fails, leave active_git_location=github and treat Forgejo as planned or failed rather than partially migrated.

Verification

Minimum verification for a migrated sparse bundle:

  • credentialed sparse checkout of the active Forgejo pointer;

  • sparse checkout of the GitHub fallback pointer;

  • presence of README.md, manifest.json when the bundle has one, scidex_refs.json, and artifact_locations.json;

  • readback of Substrate metadata for updateable refs;

  • Prism route checks for each review URL;

  • S3 list/head check for the declared prefix when S3 is listed.

The current tool performs Git sparse checkout verification. A follow-up task tracks recursive manifest comparison and S3 verification.