SPEC-064 — Schema Registry Completion

Self-describing schema_registry table; meta-schema bootstrap and validator wiring.

Source: docs/design/spec-064-schema-registry-completion.md

SPEC-064: Schema registry completion

Field Value
Status Shipped
Owner kris.ganjam@gmail.com
Date 2026-05-01
Pillar Cross-cutting
Implementation #443
Target repos: scidex-substrate (this doc, all schema JSON files, bootstrap loader, codegen). scidex-prism (consumes generated TypeScript types — separate PR).
Masters: SPEC-001 §6 (schema registry as artifact type), SPEC-009 (versioning rules), SPEC-031 (validator that depends on schemas being available).

Adjacent context:

  • src/scidex_substrate/schema_registry/default_schemas/ ships only message.json and skill.json today. Every other type relies on the v1 database having been seeded — substrate boots degraded if v1 DB is unavailable or empty.

  • 22+ artifact types exist across the codebase: hypothesis, paper, wiki_page, comment, signal, mission, dataset, debate_session, pantheon_debate, persona, agent, market, collider, arena, gap, peer_review (SPEC-055), notebook (SPEC-058), reproducibility_attestation (SPEC-056), curation_decision (SPEC-054), evidence_link (SPEC-053), flag/moderation_decision/appeal (SPEC-068).

  • scripts/export_typescript_types.py exists and consumes only the two shipped schemas. Prism’s Artifact* types are partly hand-written.

  • north-star §6 “Empty tables created ‘for later’”: v1 anti-pattern was registering empty tables. v2 corollary: don’t register a schema for a type with no producer. SPEC-064 ships schemas only for types whose owning SPEC reaches implementation in this release train.

TL;DR

Anchored against north-star.md §4 “World model = ALL artifacts”: if every persistable thing is an artifact, every artifact type needs a published, machine-readable schema. SPEC-064 lands all canonical schemas in default_schemas/ as JSON Schema draft-07, makes startup upsert the registry from those files (degraded → graceful), enforces validation at every scidex.create, and extends the codegen script to generate Prism types from the full set. Net effect: substrate is self-describing and Prism types are 1:1 with substrate reality, eliminating an entire class of drift bugs.

1. Problem statement

Three interlocking gaps:

  1. Substrate boots degraded. On a fresh DB, schema_registry is empty. scidex.create(type='hypothesis', …) either silently skips validation or 500s. SPEC-001 §6 designed the registry as an artifact type but did not pre-populate it.

  2. No source of truth for type contracts. When a SPEC declares a new type (peer_review in SPEC-055, notebook in SPEC-058), the contract lives in prose. Two readers can disagree on whether peer_review.evidence_links is required. Without a JSON Schema in the repo, the SPEC is literally the only spec.

  3. Prism types drift. scidex-prism/src/lib/types/Artifact.ts is partly hand-typed and partly generated. New types will be hand-written, badly, and silently. The codegen exists but only feeds two schemas through.

These gaps are doctrine-load-bearing: north-star §4 makes the polymorphic substrate the model of the world. A model with no published schema is not a model; it’s a guess.

2. Decision

2.1 Ship a JSON Schema for every artifact type with a producer

Each schema lives at default_schemas/<type>.json (JSON Schema draft-07, matching existing message.json and skill.json). Single object envelope:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id":     "https://scidex.ai/schemas/hypothesis@v1",
  "title":   "hypothesis",
  "type":    "object",
  "version": "v1",
  "mutability":  "mutable_with_history",     // SPEC-001 §5.3
  "id_strategy": "h-<rand12>",               // SPEC-001 §5.1
  "search": {                                 // SPEC-005 §5
    "searchable": true,
    "embedding_recipe": "title || '\\n' || content.mechanism || '\\n' || content.target_gene",
    "tsvector_recipe":  "title || ' ' || content.mechanism || ' ' || content.target_gene",
    "boosts": {"title": 2.0, "content.target_gene": 1.5}
  },
  "lifecycle": {                              // SPEC-001 §11
    "states":  ["proposed","investigating","supported","contradicted","promoted","archived"],
    "initial": "proposed"
  },
  "properties": {
    "title": {"type":"string","minLength":1,"maxLength":300},
    "content": {
      "type": "object",
      "properties": {
        "mechanism":     {"type":"string","maxLength":4000},
        "target_gene":   {"type":"string","maxLength":80},
        "evidence_for":  {"type":"array","items":{"$ref":"#/$defs/Ref"}}
      },
      "required": ["mechanism","target_gene"]
    }
  },
  "required": ["title","content"]
}

The full list of schemas this SPEC ships:

Type Mutability Owner
hypothesis.json mutable_with_history hypothesis pipeline
paper.json immutable corpus ETL (SPEC-029)
wiki_page.json mutable_with_history wiki
comment.json append_only comments
signal.json append_only signals
mission.json mutable_with_history SPEC-027
dataset.json superseded_only datasets
debate_session.json mutable_with_history classic debates
pantheon_debate.json mutable_with_history SPEC-028, SPEC-051
persona.json mutable_with_history SPEC-017, SPEC-051
agent.json mutable_with_history SPEC-017, SPEC-051
market.json mutable_with_history SPEC-018
collider.json mutable_with_history SPEC-022
arena.json mutable_with_history SPEC-032
gap.json mutable_with_history knowledge gaps
peer_review.json append_only SPEC-055
notebook.json mutable_with_history SPEC-058
reproducibility_attestation.json append_only SPEC-056
curation_decision.json append_only SPEC-054
evidence_link.json append_only SPEC-053
flag.json, moderation_decision.json, appeal.json append_only SPEC-068

Total: 23 schemas. Existing message.json and skill.json are upgraded to the same envelope.

The “no schema without a producer” rule from north-star §6 is honoured: all referenced SPECs are landing in the same release train. Schemas for types whose owning SPEC is not yet drafted (e.g. team from SPEC-051 §8) wait.

2.2 Bootstrap: upsert on substrate startup

# src/scidex_substrate/schema_registry/bootstrap.py
def bootstrap_schema_registry(db):
    for path in DEFAULT_SCHEMAS_DIR.glob("*.json"):
        schema = json.loads(path.read_text())
        validate_schema_envelope(schema)
        db.execute("""
            INSERT INTO schema_registry (artifact_type, version, schema_json, source, updated_at)
            VALUES (%s, %s, %s::jsonb, 'default_schemas', NOW())
            ON CONFLICT (artifact_type, version)
            DO UPDATE SET schema_json = EXCLUDED.schema_json, updated_at = NOW()
            WHERE schema_registry.source = 'default_schemas'
        """, (schema["title"], schema["version"], json.dumps(schema)))

The WHERE source='default_schemas' clause means operator-curated overrides (source='operator') are not clobbered. Idempotent and safe to run on every restart.

/health/deep reports schema_registry: { count: 23, missing: [] }.

2.3 Validation enforced at every scidex.create

scidex.create already looks up the schema; today it returns schema_unknown for missing types. After SPEC-064:

  • All 23 types have a schema → no schema_unknown for in-tree write paths.

  • Validator runs jsonschema.validate(content, schema['properties']['content']) and returns SPEC-001 §10 structured errors with path/constraint/value on failure.

  • Failures are 400, not retryable. scidex.batch (SPEC-062) surfaces them in per-op error.

Legacy rows that no longer satisfy strict validation stay readable; only writes are gated. A drift-scanner cron (SPEC-048) lists drifted rows for the curation loop.

2.4 Codegen: TypeScript types for Prism

Extend scripts/export_typescript_types.py to walk every schema, run json-schema-to-typescript over each, and write to scidex-prism/src/lib/types/generated/. One .ts per schema. An index.ts re-exports a discriminated union Artifact = Hypothesis | Paper | … keyed on type.

Generator runs as a CI step in scidex-substrate; output committed to scidex-prism via the substrate-pull pipeline (SPEC-014). Drift becomes a CI failure, not a runtime bug.

3. Schema / verb changes (concrete)

  • New files: 21 schemas under default_schemas/ (23 total minus 2 pre-existing).

  • Updated files: message.json, skill.json upgraded to the new envelope.

  • Bootstrap loader: new schema_registry/bootstrap.py invoked from api/app.py startup.

  • Codegen: scripts/export_typescript_types.py extended (~100 LOC) to iterate the directory.

  • Tests: tests/schema_registry/test_default_schemas_load.py — every schema parses and registers. Per-type test_<type>_create_validates.py round-trips a fixture.

  • No new tables. schema_registry already exists per SPEC-001 §6.

  • Health: /health/deep gains a schema_registry block.

4. Acceptance

  1. Boot from empty DB. Drop schema_registry, restart. Repopulated from disk; /health/deep reports count = 23, missing = [].

  2. Validation engaged for every type. scidex.create(type='peer_review', content={…valid…}) succeeds; missing required field returns 400 with path='/content/verdict', constraint='required'.

  3. Operator override preserved. Insert a schema_registry row with source='operator' for hypothesis. Restart. The operator row is not overwritten.

  4. Codegen produces compilable TypeScript. Output passes tsc --noEmit. Artifact union includes all 23 variants.

  5. Round-trip fixture per type. Each type ships tests/fixtures/<type>_valid.json; create + get round-trip succeeds.

  6. Drift detection. A type referenced in code but missing from the registry surfaces under schema_registry.missing in /health/deep.

  7. No silent skips. No scidex.create path bypasses validation. Static check: every call site either passes through the validator or is tagged # noqa: SCH001 with inline justification.

5. Rollout

Phase 1 (1 large PR): all 23 schemas + bootstrap loader + tests. Strict validation off by default behind SCIDEX_STRICT_SCHEMA_VALIDATION=0. One week soak with warnings logged but writes accepted, allowing time to fix legacy data.

Phase 2 (1 PR, S): flip default to =1. Document the flip in SPEC-019 cutover runbook.

Phase 3 (1 PR, M): codegen integrated into substrate-pull pipeline. Prism PR consumes generated types; hand-typed Artifact*.ts files deleted in the same Prism PR.

Phase 4 (per type): when a future SPEC adds a new type, the SPEC must include the schema file in the same PR as the first producer. Reviewer checklist enforces.

Compatible with cutover-runbook: existing writes against prod continue because Phase 1 doesn’t enforce. Phase 2’s flip is a planned date with a rollback flag.

6. Open questions

  • JSON Schema vs Pydantic-as-source. Several types have Pydantic models. Should those be the source and JSON Schema be derived? Decision: JSON Schema is the source (language-neutral; Prism reads directly). Where Pydantic exists, CI auto-checks against the JSON Schema.

  • Type aliasing across cutover. v1’s actor_type strings differ from v2’s agent.kind per SPEC-051. agent.json documents both until the back-compat alias column is removed (SPEC-051 §4.2 timeline).

  • Per-type lifecycle vocabulary. lifecycle.states must match what curation SPECs expect. SPEC-054 owns the state vocabulary; coordinate during PR review.

  • Multilingual content. All schemas English-only today. If non-English papers arrive, do we add lang per type or universal content.lang? Defer; not required for cutover.

  • Schema versioning across upgrades. SPEC-009 owns v1 → v2 evolution. SPEC-064 ships only v1.

  • External consumers. Once published, oxa.dev mirrors (SPEC-021) and federated readers may consume them. $id already points at https://scidex.ai/schemas/<type>@<v>; serving it is a small follow-up.

Work Log

2026-05-14 — SPEC-064 §5 Phase 1 implementation

Commit: 27ce9c5 — [Cross-cutting][SPEC-064 §5 Phase 1] SCIDEX_STRICT_SCHEMA_VALIDATION=0 + test expansion

What landed:

  1. SCIDEX_STRICT_SCHEMA_VALIDATION flag wired into scidex.create (handler.validators + TypedCreateValidator paths). Default 0 — failures log a warning but allow writes through (Phase 1 soak mode). Set to 1 to enforce and block.

  2. tests/service/test_schema_registry_bootstrap.py expanded from 10 hardcoded SPEC-064 types to all 29 canonical types currently in default_schemas/ (after filtering gap_mission — the mission.json filename declares type: gap_mission internally, a pre-existing mismatch tracked in SPEC-064 §6 open questions).

What was already done (prior commits on main):

  • All 30 schema JSON files present in default_schemas/ (ca04e7b, #443)

  • seed_default_schemas() idempotent upsert in schema_registry/__init__.py

  • Bootstrap loader invoked from api/app.py at startup (SCIDEX_AUTO_SEED_SCHEMAS=1)

  • /health/deep reports schema seed status (613b09f)

  • Meta-schema, envelope validation, and lifecycle block tests

Still open (not in this commit):

  • Phase 2: flip SCIDEX_STRICT_SCHEMA_VALIDATION default to =1 + SPEC-019 cutover doc

  • Phase 3: codegen integration into substrate-pull pipeline

  • Phase 4: per-type schema-in-PR rule for future SPECs

  • mission.json filename↔type gap_mission mismatch resolution