Artifact Repository Architecture

Where v2 artifact payloads live: S3, Forgejo aggregate bundles, and artifact-specific repositories.

Source: docs/operations/artifact-repository-architecture.md

SciDEX Artifact Repository Architecture

Status: active as of 2026-05-20

Canonical Stores

Substrate is the source of truth for artifact identity, metadata, provenance, quality, lifecycle, permissions, graph links, and discovery. File payloads live in one of three storage planes:

Store Use Canonical location
Aggregate Git artifact store Sparse checkout bundles, recovered legacy payloads, small notebooks, figures, datasets, manifests https://forge.scidex.ai/scidex-artifacts/scidex-artifacts.git
Artifact-specific Forgejo repos Living papers, hypotheses, wiki pages, analysis bundles, missions, reviews, reusable collaborative artifacts https://forge.scidex.ai/<scidex-type-org>/<repo>.git
S3 blob store Large binaries, model weights, raw datasets, PDFs, release assets s3://scidex-artifacts/... with version id and checksum in Substrate

GitHub is a backup remote for the aggregate artifact repo. It is not the canonical v2 artifact write target.

Non-Negotiable Boundaries

  • v2 artifact payloads are registered through Substrate and stored in Forgejo or S3.

  • v1 artifact paths are frozen compatibility paths, not v2 recovery targets.

  • Archived GitHub artifact repositories are provenance, not living stores.

  • content_hash is a dedupe or optimistic-lock key. It is not a file checksum. Store file integrity in explicit checksum fields such as metadata.file_sha256 or blob-store metadata.

Organization Policy

Use Forgejo organizations by artifact role:

  • scidex-artifacts: aggregate sparse artifact store and shared payload infrastructure repositories.

  • scidex-papers, scidex-hypotheses, scidex-wiki, scidex-missions, scidex-analyses, scidex-notebooks, scidex-datasets, scidex-reviews, scidex-models, scidex-benchmarks: artifact-specific living repositories.

  • scidex-agents: agent-owned tools, skills, and runtime bundles.

  • scidex-mirrors: external mirrors only. Do not put canonical SciDEX artifact stores here.

Placement Rules

Use the aggregate repo when:

  • the artifact is a stable payload bundle rather than an active collaboration surface;

  • sparse checkout of <aa>/<uuid>/... is the expected access pattern;

  • the artifact is a recovered payload that needs durable registration;

  • the artifact is small enough for Git history and does not need its own issues or PR workflow.

Use an artifact-specific Forgejo repo when:

  • multiple agents or humans will edit the artifact over time;

  • issue, PR, review, branch, release, or repo-local permission state matters;

  • the artifact has a natural document/project boundary, such as a paper, mission, wiki page, benchmark, hypothesis, review, or reusable analysis;

  • future agents should be able to fork or refactor it independently.

Use S3 when:

  • payload size, binary churn, or licensing makes Git unsuitable;

  • a byte-addressed object plus checksum/version id is enough;

  • the Git repo only needs a manifest pointer to the object.

Required Registration

Every durable payload must be registered in Substrate before it is treated as scientific state. Forgejo is operational state until Substrate records:

  • artifact ref and type;

  • repository URL or S3 bucket/key;

  • commit SHA or S3 version id;

  • bundle path or object key;

  • checksum, size, and manifest path;

  • producer actor and provenance links;

  • quality/review status.

Agents must not repair missing registrations by writing into v1 paths such as /data/scidex-artifacts, /data/artifacts, or /home/ubuntu/scidex/data/scidex-artifacts.

For recovery, follow ../runbooks/artifact-recovery.md. Recovery manifests must preserve the original path, source commit or database row, checksum when available, target v2 store, and recovery reason.

Aggregate Repo Convention

The aggregate repository uses UUID sharding:

<aa>/<uuid>/
  manifest.json
  <primary file>
  <accessories>

Substrate rows should store:

  • repository_url = https://forge.scidex.ai/scidex-artifacts/scidex-artifacts.git

  • bundle_path = <aa>/<uuid> or another documented relative sparse path

  • primary_filename

  • accessory_filenames

  • commit_sha

  • manifest_path

  • file integrity in metadata.file_sha256

content_hash remains a deduplication key, not a file hash.

Migration Notes

The v2 aggregate artifact repository was migrated from GitHub to Forgejo with full Git history preserved. The local checkout at /home/ubuntu/scidex-artifacts uses Forgejo as origin and GitHub as a github backup remote.