Designing a Schema

How to add a new artifact type: JSON Schema authoring, envelope fields, auto-registration, which verbs come for free.

Source: docs/tutorials/designing-a-schema.md

Designing a new artifact type

The substrate is polymorphic by design: every persistable object is an Artifact{type, id, version} and the ~17 core verbs (scidex.get, list, search, create, update, comment, link, signal, subscribe, …) work across every registered type. Adding a new type is therefore the cheapest way to extend the substrate — once the schema is in place, you get those verbs for free.

This tutorial adds a hypothetical assay_result type: a record of one in-vitro or organoid assay, linked to the hypotheses it tests. It’s a realistic example that exercises every part of the schema model: content fields, optimistic locking, link predicates, signals, lifecycle states, and search.

Read SPEC-001 §5 and SPEC-064 before opening a PR for a new type. For changes to an existing type, also read SPEC-009.

Before you add a type

Use a new artifact type only when the object needs a distinct contract: its own schema, lifecycle, validation, search recipe, or link/signal vocabulary. If the object is only a presentation variant, a tag, or a one-off payload, prefer fields on an existing artifact type or a linked child artifact.

Checklist:

  • The type has a clear user-facing name and purpose.

  • It cannot be represented cleanly by an existing type.

  • The minimum required fields are small.

  • Payload bytes, if any, have an approved S3 or Forgejo storage plan.

  • The type declares lifecycle, search, links, and signals only where needed.

  • The PR includes tests and a Prism/client visibility plan if users will see the type.

This replaces the v1 pattern of scaffolding a new Python module, migration, template, and route for every artifact kind. In v2, schemas and polymorphic verbs are the default; custom routes are the exception.

The two-step recipe

The substrate’s contract from AGENTS.md:

  1. Create src/scidex_substrate/schema_registry/default_schemas/<type>.json.

  2. Restart the substrate; the schema is auto-registered.

This is true for the schema — the data contract. But to make polymorphic verbs actually return rows, you also need a Postgres handler (and usually a migration for the per-type table). The full picture is:

  1. Write the JSON schema file.

  2. Add a HandlerSpec registration in src/scidex_substrate/skill/handlers.py.

  3. Ship a migration in src/scidex_substrate/migrations/ if you need a dedicated per-type table (or reuse the generic artifacts table for very simple types).

Steps 2 and 3 are what let scidex.list, scidex.search, and scidex.get find your rows. Without them, validation works but reads return empty pages.

1. The schema file

Create src/scidex_substrate/schema_registry/default_schemas/assay_result.json:

{
  "type": "assay_result",
  "schema_version": 1,
  "id_strategy": {
    "format": "asy-{rand12}",
    "stable": true,
    "forbidden_chars": [":", "@"]
  },
  "lock_mode": "content_hash",
  "mutability": "mutable_with_history",
  "content_schema": {
    "type": "object",
    "required": ["title", "assay_kind", "outcome"],
    "properties": {
      "title": {
        "type": "string",
        "minLength": 1,
        "maxLength": 300
      },
      "summary": {
        "type": "string",
        "maxLength": 4000,
        "description": "Short human-facing description of the assay setup."
      },
      "assay_kind": {
        "type": "string",
        "enum": [
          "biochemical",
          "cell_based",
          "organoid",
          "animal",
          "clinical"
        ],
        "description": "What system was tested in."
      },
      "outcome": {
        "type": "string",
        "enum": ["positive", "negative", "inconclusive", "withdrawn"],
        "description": "Top-line result classification."
      },
      "effect_size": {
        "type": "number",
        "description": "Standardized effect (e.g. Cohen's d). Optional."
      },
      "p_value": {
        "type": "number",
        "minimum": 0,
        "maximum": 1,
        "description": "Reported significance. Optional."
      },
      "tested_hypothesis_refs": {
        "type": "array",
        "items": {"type": "string"},
        "description": "Refs (type:id) of hypotheses this assay tests."
      },
      "reagents": {
        "type": "array",
        "items": {"type": "string"},
        "description": "Reagent identifiers (CAS, vendor SKU, …)."
      },
      "epistemic_tier": {
        "type": "string",
        "enum": ["T0", "T1", "T2", "T3", "T4", "T5"],
        "default": "T0"
      }
    }
  },
  "links": {
    "tests":             {"to_types": ["hypothesis"]},
    "uses_reagent":      {"to_types": ["reagent"]},
    "supersedes":        {"to_types": ["assay_result"]},
    "supports":          {"to_types": ["hypothesis"]},
    "contradicts":       {"to_types": ["hypothesis"]}
  },
  "signals": {
    "vote": {
      "values": [-1, 1],
      "aggregation": "replace"
    },
    "rank": {
      "dimensions": ["rigor", "reproducibility", "novelty"],
      "value_range": [0, 1],
      "aggregation": "replace"
    }
  },
  "lifecycle": {
    "states": ["active", "deprecated", "cold", "superseded", "quarantined"],
    "initial": "active"
  },
  "search": {
    "searchable": true,
    "embedding_recipe": "title + '\\n' + summary",
    "tsvector_recipe": "title || ' ' || summary",
    "boosts": {"title": 2.0, "summary": 1.0}
  },
  "atproto_nsid": "com.scidex.assay_result"
}

Every key here is meaningful — the rest of this section explains them.

id_strategy

How the substrate generates ids for new artifacts. The format template uses two recognized tokens:

  • {rand12} — a URL-safe 12-character random suffix (asy-7c1c91f2b3a9).

  • {slug} — a slug derived from the content (used by paper, skill, tool).

stable: true means the id, once minted, never changes. forbidden_chars forbids : and @ because both are used by the canonical ref string form "<type>:<id>[@<content_hash>]".

lock_mode and mutability

These two fields define the optimistic-locking and history semantics the substrate applies on writes.

lock_mode:

  • content_hash (typical) — updates must present a base_content_hash to prove they’re editing the version they read.

  • none — updates always overwrite. Use only for trivial counter-like types where conflicts are uninteresting.

mutability (the four states from skill/types.py):

  • immutable — create once, never update. Papers, datasets, external citations.

  • append_only — create and append child rows; no in-place edits. Events, audit logs.

  • mutable_with_history — full versioning; every update writes a history row keyed on the prior content_hash. Hypotheses, wiki pages, schemas themselves.

  • superseded_only — updates always supersede (new id under the same logical entity); the old id stays accessible by content_hash.

For assay_result we picked mutable_with_history because the authoring biologist may correct effect sizes or reagent lists post-hoc, but the full audit trail must survive.

content_schema

A JSON Schema (Draft 2020-12) that the substrate validates content against on every create / update. Pydantic in the verb layer guards field types; this schema enforces the substrate-side invariants and is what other agents see when they call scidex.schema(type='assay_result').

Two non-obvious rules from SPEC-064:

  • required lists the minimum set that must be present on every successful write. Pick the smallest set you can; everything else should be optional.

  • default values are honoured by the substrate on create — see epistemic_tier’s default of "T0" in this schema.

links

Declares the typed predicates this type can emit, with their allowed target types. The substrate’s scidex.link verb consults this map and rejects edges whose (predicate, to_type) isn’t declared. Reverse edges (incoming links) are inferred at query time — you don’t declare “papers are cited by hypotheses” anywhere; scidex.list and the link indexes find the reverse direction.

signals

Signals are the polymorphic vote / rank / fund surface (SPEC-001 §10). Each artifact type declares which signal kinds apply and how they aggregate.

  • vote with {"values": [-1, 1], "aggregation": "replace"} — binary up/down vote, one per actor, last write wins.

  • rank with dimensions and a [0, 1] range — weighted multi-axis scoring.

  • fund with a currency and "sum" aggregation — token allocations.

If you don’t declare a signal kind, calls to scidex.signal for that type return validation_failed.

lifecycle

The set of states the artifact moves through, with initial as the state on create. The platform-standard states from SPEC-111 are active, deprecated, cold, superseded, quarantined — use these unless your type genuinely needs custom states. scidex.search and scidex.list filter out non-active states by default.

search

Tells the substrate how to build full-text and embedding-eligible representations. tsvector_recipe is interpolated into Postgres generated columns; embedding_recipe is what the embedding pipeline hashes and feeds to the model. The boosts map weights matched fields in lexical scoring.

atproto_nsid

The federation NSID under SPEC-021. Use the com.scidex.<type> convention.

2. The polymorphic envelope you get for free

Every artifact response is wrapped in ArtifactEnvelope. The envelope’s shape is type-uniform — every artifact, whatever its content schema, comes back with the same outer fields:

class ArtifactEnvelope(BaseModel):
    ref: Ref                                  # type + id (+ optional version)
    title: str
    type: str
    created_at: datetime | None
    updated_at: datetime | None
    created_by: ActorRef | None
    is_latest: bool
    superseded_by: Ref | None
    content_hash: str | None
    quality_score: float | None
    schema_version: int | None
    provenance_chain: list[ProvenanceStep]
    content: dict[str, Any]                   # your content_schema-validated payload
    links: list[Link] | None
    signals: SignalSummary | None
    epistemic_tier: EpistemicTier | None
    lifecycle_state: Literal["active", "deprecated", "cold", "superseded", "quarantined"] | None

content is the only field whose shape is type-specific; everything else is the substrate’s contract. The is_latest / superseded_by pair lets clients tell whether they’re looking at the current version or a historical one. schema_version powers lazy migration: if the stored row is on schema_version=1 and the registry now ships 2, scidex.get runs your registered @migrator chain in-flight to upgrade the payload before returning (SPEC-009 §5.1).

3. The handler registration

The schema file by itself is enough for scidex.schema to surface your type to agents. But for scidex.list, scidex.get, and scidex.search to actually find rows, the substrate needs to know which Postgres table holds them. That happens in skill/handlers.py through a register(HandlerSpec(...)) call.

Add a block alongside the existing types (the file is one register call per type; they’re sorted alphabetically):

register(
    HandlerSpec(
        type="assay_result",
        table="assay_results",
        id_column="id",
        title_column="title",
        search_snippet_column="summary",
        id_prefix="asy",
        has_updated_at=True,
        content_hash_column="content_hash",
        create_required=[],   # SPEC-031 default validators fire from title_column
        system_defaults={"version": 1, "last_mutated_at": "__NOW__"},
        content_columns=[
            "summary",
            "assay_kind",
            "outcome",
            "effect_size",
            "p_value",
            "reagents",
            # SPEC-009: flat columns the handler pulls into envelope.content
        ],
        history_table="assay_results_history",
        history_id_column="assay_id",
        # When the content_schema has fields the per-type table does
        # NOT have columns for (predictions, evidence_refs in the
        # hypothesis case), set this so the handler reads them back
        # from artifacts.metadata.content.
        supplement_from_artifacts=False,
    )
)

Read the existing entries for hypothesis (line ~1300 in handlers.py) and paper (line ~1344) to see the full set of options.

4. The migration

If you need a dedicated per-type table (the common case), add a SQL migration file. The runner picks up everything in src/scidex_substrate/migrations/ whose filename starts with the next available number. Run scripts/next_migration_number.sh to get it.

-- src/scidex_substrate/migrations/0235_assay_results.sql

CREATE TABLE IF NOT EXISTS assay_results (
    id              TEXT PRIMARY KEY,
    title           TEXT NOT NULL,
    summary         TEXT,
    assay_kind      TEXT NOT NULL
                       CHECK (assay_kind IN (
                         'biochemical', 'cell_based', 'organoid',
                         'animal', 'clinical'
                       )),
    outcome         TEXT NOT NULL
                       CHECK (outcome IN (
                         'positive', 'negative', 'inconclusive', 'withdrawn'
                       )),
    effect_size     DOUBLE PRECISION,
    p_value         DOUBLE PRECISION CHECK (p_value BETWEEN 0 AND 1),
    reagents        JSONB DEFAULT '[]'::jsonb,
    content_hash    TEXT NOT NULL,
    version         INTEGER NOT NULL DEFAULT 1,
    lifecycle_state TEXT NOT NULL DEFAULT 'active',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    last_mutated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_by      TEXT REFERENCES actors(id),
    -- SPEC-005: tsvector generated column powers scidex.search lexical mode.
    search_vector   TSVECTOR GENERATED ALWAYS AS (
        setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
        setweight(to_tsvector('english', coalesce(summary, '')), 'B')
    ) STORED
);

CREATE INDEX IF NOT EXISTS idx_assay_results_search
    ON assay_results USING GIN (search_vector);

CREATE INDEX IF NOT EXISTS idx_assay_results_created_at
    ON assay_results (created_at DESC);

CREATE INDEX IF NOT EXISTS idx_assay_results_lifecycle
    ON assay_results (lifecycle_state)
    WHERE lifecycle_state <> 'active';

CREATE TABLE IF NOT EXISTS assay_results_history (
    history_id      BIGSERIAL PRIMARY KEY,
    assay_id        TEXT NOT NULL REFERENCES assay_results(id) ON DELETE CASCADE,
    version         INTEGER NOT NULL,
    content_hash    TEXT NOT NULL,
    content         JSONB NOT NULL,
    written_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    written_by      TEXT REFERENCES actors(id),
    UNIQUE (assay_id, version)
);

A couple of conventions from the rest of the codebase that you should keep:

  • Always CREATE TABLE IF NOT EXISTS. The migration runner is idempotent on file (it tracks applied filenames in substrate_schema_migrations), but defensive SQL means a backup-restored DB doesn’t trip on re-application.

  • search_vector as a STORED generated column. The tsvector_recipe in the JSON schema is your documentation of how the vector should be built; the migration is the implementation. Keep them in sync.

  • A separate history table when mutability=mutable_with_history. The scidex.update write path writes the post-image to the per-type table and the pre-image to the history table inside one transaction.

5. What you get for free, what you don’t

Once the schema, handler, and migration land, these verbs work immediately for the new type with no further code:

Verb What it does for the new type
scidex.schema(type='assay_result') Returns the JSON schema.
scidex.types Lists your type alongside the others.
scidex.list(type='assay_result') Paginated read.
scidex.get(ref='assay_result:asy-…') Single artifact read with full envelope.
scidex.search(types=['assay_result'], query=…) Lexical + structured search. Hybrid/semantic light up once the embedding pipeline backfills embeddings.
scidex.create(type='assay_result', …) Validate against content_schema, mint id, write the row + emit assay_result.created.
scidex.update(ref=…, …) Lock-token-protected mutate, append to history table, emit assay_result.updated.
scidex.comment(ref=…, body=…) Append a comment to the new artifact.
scidex.link(from=…, predicate='tests', to='hypothesis:h-…') Link create with links map enforcement.
scidex.signal(ref=…, kind='vote', value=1) Vote signal with signals map enforcement.

What does not come for free:

  • Type-specific business logic. If creating an assay_result should trigger a calibration update on the tested hypothesis, that needs a verb (or a worker subscribing to events).

  • Specialized queries. scidex.list filters via JSONLogic; complex joins (e.g. “all assay results whose tested hypothesis is currently in the supported lifecycle state”) need a dedicated verb.

  • Embedding backfill. New rows get embeddings via the backfill pipeline (see scripts/scidex_search_backfill_embeddings.py). Until the first run, mode='semantic' falls back to lexical with a meta.fallback='lexical' marker — read verbs/search.py for the auto-mode resolution rules.

  • Validators beyond JSON Schema. Cross-field checks (e.g. “p_value required when outcome is positive”) go in src/scidex_substrate/skill/validators/. The default-validator registry (SPEC-031) wires title_min_length and title_no_html automatically when you set title_column.

6. Verify the registration

After restart, the auto-seeder copies your schema file into the schema_registry table:

psql "$SCIDEX_DSN" -c \
  "SELECT type, schema_version, is_current FROM schema_registry WHERE type='assay_result';"

       type        | schema_version | is_current
-------------------+----------------+-----------
 assay_result      |              1 | t

Verify the verbs see it:

curl -s http://127.0.0.1:8200/api/scidex/types -X POST -d '{}' \
  -H 'content-type: application/json' | jq '.types[] | select(. == "assay_result")'

"assay_result"

And exercise an end-to-end create + read round-trip:

# Mint a dev JWT (SCIDEX_DEV_AUTH=1 only)
curl -s -c cookies.txt -X POST \
  http://127.0.0.1:8200/auth/dev/login \
  -H 'content-type: application/json' \
  -d '{"email": "you@example.com"}'

# Create
curl -s -b cookies.txt -X POST \
  http://127.0.0.1:8200/api/scidex/create \
  -H 'content-type: application/json' \
  -d '{
    "type": "assay_result",
    "content": {
      "title": "LRRK2-IN-1 in iPSC-derived dopaminergic neurons",
      "summary": "10 μM LRRK2-IN-1 reduces phospho-Rab10 60%.",
      "assay_kind": "cell_based",
      "outcome": "positive",
      "effect_size": 0.83,
      "p_value": 0.003
    }
  }' | jq

# List
curl -s http://127.0.0.1:8200/api/scidex/list \
  -X POST \
  -H 'content-type: application/json' \
  -d '{"type": "assay_result", "limit": 5}' | jq

7. Tests you owe the substrate

Per SPEC-007:

  • A schema-conformance test: load the JSON schema, validate a representative happy-path content and one of each invalid shape. Lives under tests/unit/schema_registry/.

  • A handler test: assert that register(HandlerSpec(...)) succeeds and that get_handler('assay_result') returns the same spec.

  • A migration test: apply the migration to a clean _test database and assert tables / indexes exist with the expected columns.

  • A service-layer test for each verb you exercised in §6 (scidex.create, scidex.list, scidex.get, scidex.search) against your new type.

The pre-existing tests for hypothesis and paper are the canonical references — copy their shape.

8. References