SPEC-001 — SciDEX Polymorphic Substrate

Field	Value
Status	Draft v1
Owner	kris.ganjam@gmail.com
Date	2026-04-28
Pillar	Cross-cutting
Predecessor	`docs/design/mcp_artifact_edits.md` (Phase 1 — superseded)
Companion specs	002 Forge · 003 Prism · 004 Rate limits · 005 Semantic search · 006 Migration · 007 Tests · 008 Observability · 009 Schema evolution · 010 Identity · 011 Federation · 012 CI/CD · 013 Skills · 014 Substrate repo · 125 API styles
Implementation repo	`SciDEX-AI/scidex-substrate` (new — per SPEC-014)

2026-05-20 amendment — agent unification. The substrate renamed the actors table to agents and pluralized its verb namespaces (actor.* → agents/*, hypothesis.* → hypotheses/*, etc.) in the 2026-05 unification session (session note, PRs #2009 → #2014). Where this spec still says actor_id / actor.X as a concept (in JWT claims, the Context object, identity stamping, idempotency keys), the concept is unchanged — that is the agent identity model and is preserved verbatim. Where this spec sketches URL paths or verb names (e.g. scidex.message.* in §13), the canonical form is now plural (messages/*, agents/*). SPEC-125 captures the REST-vs-RPC rationale and naming-hygiene rule.

TL;DR

Collapse SciDEX’s 841 HTTP routes + dual MCP servers into a small set of polymorphic verbs (~17 core + ~7 action) covering ~80% of agent operations. Single Python source-of-truth: HTTP routes and MCP tools are both generated from typed Pydantic functions. Agents get structured errors, recoverable conflicts, idempotency, server-stamped identity, and a reactive event substrate. API-only — no storage migration.

1. Goals & non-goals

1.1 Goals

One artifact model. Every persistable object — hypothesis, paper, wiki page, comment, vote, KG edge, debate session, schema, tool — is an Artifact{type, id, version}.
Polymorphic verb surface. ~17 core verbs work across all artifact types. Adding a type requires zero new verb code.
Generated tooling. @verb-decorated Pydantic functions produce HTTP routes and MCP tool definitions in one pass.
Agent-first ergonomics. Validation errors carry path/constraint/value. Conflicts return full current content + JSONPatch + clean-replay flag.
Reactive substrate with multi-modal delivery. Agents subscribe (push/SSE) for live work, poll on cron, or use a durable message artifact type for agent-to-agent directives that survive restart.
Self-describing. Schema registry is itself a registered artifact type.

1.2 Non-goals (this spec)

Per-actor rate-limit policy — see SPEC-004.
Search engine selection / pgvector migration — see SPEC-005.
Frontend rewrite — see SPEC-003.
Tool runtime design — see SPEC-002.
Migration mechanics — see SPEC-006. v2 is a clean cutover from v1; storage consolidation happens at migration time.

2. Glossary

Term	Meaning
Substrate	The SciDEX backend exposed as a polymorphic artifact store with events.
Artifact	Any persistable, addressable, schema-validated object.
Verb	A polymorphic operation (`get`, `update`, `subscribe`, …) that applies across types.
Ref	`{type, id, version?}` — canonical string form `"<type>:<id>[@<content_hash>]"`.
Signal	A named, dimensioned, weighted scalar attached to an artifact (votes, ranks, fund commitments, calibration scores).
Link / edge	A typed predicate connecting two refs.
Lock token	The `content_hash` an updater presents to prove they’re editing the version they read.
Context	Server-stamped per-request identity, session, delegation chain, idempotency key.

3. Architecture

┌────────────────────────────────────────────────────────────────┐
│  Agents (Claude / Codex / Orchestra) │ Prism (frontend)         │
└───────────┬───────────────────────────────────┬───────────────┘
            │ MCP (stdio/SSE)                   │ HTTP/SSE
            ▼                                   ▼
   ┌──────────────────────────────────────────────────┐
   │  Polymorphic Verb Layer (FastAPI + Pydantic)      │   ← generated
   │  scidex.skill.{get,list,search,create,update,…}   │
   └──────────────────┬───────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────┐
   │  Service Layer (typed Python)                     │
   │  - dispatch by artifact_type                      │
   │  - server-stamped identity & delegation chain     │
   │  - validation against schema_registry             │
   │  - optimistic locking & three-way merge hints     │
   │  - event emission (LISTEN/NOTIFY)                 │
   └────────┬─────────────────────────────────┬──────┘
            ▼                                 ▼
   ┌─────────────┐                 ┌────────────────────┐
   │ artifacts + │                 │ events table       │
   │ companions  │                 │ + pg_notify        │
   └─────────────┘                 └────────────────────┘

The verb layer and the MCP server are both outputs of the service layer. There is no parallel implementation.

3.1 Generation pipeline

# scidex/skill/verbs/get.py
from scidex.skill.framework import verb, Context, Ref

class GetIn(BaseModel):
    ref: Ref
    include_content: bool = True
    include_links: bool = False
    include_signals: bool = False

class GetOut(BaseModel):
    ref: Ref
    artifact: ArtifactEnvelope
    links: list[Link] | None = None
    signals: SignalAggregates | None = None

@verb(read=True)
def get(args: GetIn, ctx: Context) -> GetOut:
    ...

The @verb decorator registers the function in three places at import time:

FastAPI route table → POST /api/scidex/get.
MCP tool registry → tool scidex.get with input schema derived from GetIn.model_json_schema().
Internal Python callable → scidex.skill.get(args, ctx) for in-process callers.

No drift is possible.

4. Identity model

4.1 JWT claims

{
  "sub":               "<actor-uuid>",
  "actor_kind":        "human" | "ai_local" | "ai_external" | "ai_swarm",
  "permissions":       ["viewer" | "contributor" | "reviewer" | "admin"],
  "session_id":        "<session-uuid>",
  "parent_session_id": "<session-uuid>" | null,
  "delegation_chain": [
    { "actor_id": "...", "session_id": "...", "task_id": "..." }
  ],
  "iat": 0, "exp": 0
}

4.2 Context object (server-stamped, request-scoped)

@dataclass
class Context:
    actor:             ActorRef             # id, kind, permissions
    session:           SessionRef           # id, parent_id
    delegation_chain:  list[DelegationStep]
    request_id:        UUID                 # for log correlation
    idempotency_key:   str | None
    task_id:           str | None           # from JWT or env (Orchestra)
    now:               datetime             # frozen per-request for determinism
    db:                Connection           # request-scoped via ContextVar

The DB connection is fetched from a request-scoped ContextVar, not thread-local. This is enforced — see the existing PG pool fix (2026-04-21).

4.3 Server-stamping rule

Agents never pass identity in tool args. The MCP/HTTP layer reads JWT once at handshake, materializes Context, and stamps every write into db_write_journal and per-type history tables. Bad writes are traceable to the full (human → orchestra → claude_subagent) lineage.

4.4 Identity extensibility

The JWT model is the v1 substrate session token. Future identity providers (wallet-signed challenges, DIDs, ATProto-issued credentials) plug in at the bootstrap layer — they mint a JWT, the substrate’s verbs treat all JWTs uniformly. Per-action cryptographic signatures (for high-stakes governance/funding writes) are reserved as WriteEnvelope.additional_signatures from v1; the field is empty until validators that need it land. See SPEC-010 for the long-term identity architecture and SPEC-011 for cross-network publication.

5. Artifact model

5.1 Reference

class Ref(BaseModel):
    type: str               # registered artifact type
    id: str                 # opaque, type-specific
    version: str | None     # content_hash; null = latest

String form: "<type>:<id>[@<content_hash>]" — example: "hypothesis:h-7a9c642b@sha256:abc123…".

The schema registry’s id_strategy declares the id format and forbids : and @ characters in ids (reserved as ref delimiters).

5.2 Common envelope

Every artifact, regardless of type, exposes:

class ArtifactEnvelope(BaseModel):
    ref:               Ref
    title:             str
    type:              str
    created_at:        datetime
    updated_at:        datetime
    created_by:        ActorRef
    is_latest:         bool
    superseded_by:     Ref | None
    content_hash:      str
    quality_score:     float | None
    provenance_chain:  list[ProvenanceStep]
    content:           dict           # type-specific, validated against schema
    links:             list[Link]     # outgoing edges (paginated; opt-in)
    signals:           SignalSummary  # aggregated votes/ranks/funds (opt-in)

Type-specific data lives entirely under content. The envelope is type-uniform.

5.3 Mutability declared per-type

Each schema declares one of:

Mode	Semantics	Examples
`immutable`	Created once, never modified.	`paper`
`append_only`	New rows only; no updates.	`comment`, `vote`, `kg_edge`, `debate_round`, `tool_call`, `signal`
`mutable_with_history`	Updates allowed; full snapshot to `_history` on each update.	`hypothesis`, `wiki_page`, `knowledge_gap`
`superseded_only`	Old version sets `is_latest=false`; new version inserted.	`analysis`, `model`, `dataset`

The update verb refuses operations the schema disallows.

6. Schema registry

6.1 Storage

CREATE TABLE schema_registry (
  id              TEXT PRIMARY KEY,    -- "hypothesis@v3"
  artifact_type   TEXT NOT NULL,
  schema_version  INTEGER NOT NULL,
  is_current      BOOLEAN NOT NULL,
  schema_json     JSONB NOT NULL,
  proposed_by     TEXT,
  approved_at     TIMESTAMPTZ,
  created_at      TIMESTAMPTZ DEFAULT NOW(),
  UNIQUE(artifact_type, schema_version)
);
CREATE INDEX ON schema_registry(artifact_type, is_current);

6.2 Schema-as-artifact

schema is itself a registered artifact type. Schema changes go through scidex.update(type="schema", id="hypothesis", patch=…). Senate can vote on them. Schemas have history, comments, supersession, and provenance — all for free via the polymorphic verbs.

Bootstrap: a hardcoded meta-schema (the JSON Schema for what a schema looks like) ships in code; all subsequent schemas live in the table.

6.3 Schema document shape

{
  "type": "hypothesis",
  "schema_version": 3,
  "id_strategy": {
    "format": "h-{uuid}",
    "stable": true,
    "forbidden_chars": [":", "@"]
  },
  "lock_mode": "content_hash",
  "mutability": "mutable_with_history",
  "content_schema": {
    "type": "object",
    "required": ["title", "target_gene", "mechanism"],
    "properties": { /* JSON Schema Draft 2020-12 */ }
  },
  "links": {
    "supports":     { "to_types": ["paper", "dataset"], "evidence_required": true, "cardinality": "many" },
    "contradicts":  { "to_types": ["hypothesis", "paper"], "evidence_required": true },
    "tests":        { "to_types": ["experiment"] },
    "derives_from": { "to_types": ["analysis", "debate_session"] }
  },
  "signals": {
    "rank":        { "dimensions": ["mechanistic_plausibility", "evidence_strength", "novelty", "feasibility", "therapeutic_potential"], "aggregation": "replace", "value_range": [0, 1] },
    "vote":        { "values": [-1, 1], "aggregation": "replace" },
    "fund":        { "currency": "scidex_token", "aggregation": "sum" },
    "calibration": { "metric": "brier", "aggregation": "append" }
  },
  "lifecycle": {
    "states": ["proposed", "investigating", "resolved", "deprecated"],
    "transitions": [["proposed","investigating"], ["investigating","resolved"], ["*","deprecated"]]
  },
  "validators": [
    "resolve_pmid_citations",
    "no_inline_pmid_outside_citation_block",
    "lifecycle_transition_allowed"
  ],
  "merge": {
    "text_fields": ["mechanism", "description"]
  }
}

The signals.* block makes voting/ranking/funding extensible — see §10.

7. Verbs

All verbs are typed Pydantic functions. All take a server-stamped Context. All return structured results with explicit error variants.

7.1 Read verbs

Verb	Args	Returns
`scidex.get`	`ref, include_content?, include_links?, include_signals?`	`ArtifactEnvelope`
`scidex.list`	`type, filter, sort, limit, cursor`	`Page<ArtifactEnvelope>`
`scidex.search`	`query, types?, filter?, mode={auto,lexical,semantic,hybrid,structured}, limit, cursor, rerank, highlight`	`Page<SearchHit>`
`scidex.history`	`ref`	`list[ArtifactEnvelope]`
`scidex.diff`	`ref_a, ref_b`	`UnifiedDiff`
`scidex.schema`	`type?, version?`	`SchemaDoc` or `list[SchemaDoc]`
`scidex.links`	`ref, direction={out,in,both}, predicates?`	`Page<Link>`
`scidex.signals`	`ref, kinds?`	`SignalAggregates`

7.2 Write verbs

Verb	Args	Returns
`scidex.create`	`WriteEnvelope (op=create)`	`WriteResult`
`scidex.update`	`WriteEnvelope (op=update, base_content_hash)`	`WriteResult` ∪ `VersionConflict`
`scidex.supersede`	`ref, replacement_ref?, reason`	`WriteResult`
`scidex.comment`	`ref, body, thread_parent?, kind`	`WriteResult`
`scidex.signal`	`ref, kind, dimension?, value, weight?, reason?`	`WriteResult`
`scidex.link`	`from, predicate, to, evidence_refs?`	`WriteResult`
`scidex.cite`	`ref, paper_pmid, span?, claim?`	`WriteResult`
`scidex.batch`	`list[WriteEnvelope], atomic, idempotency_key`	`BatchResult`

7.2.1 Write confirmation and discovery

Every write verb must return enough information for an autonomous agent to verify its own work without reading private database state:

stable ref and content/version hash;
request id and actor/runtime provenance;
any provenance refs stamped by the server;
validation warnings that were accepted but should be reviewed later;
idempotency outcome when the write reused an existing artifact.

After scidex.create, the returned ref must be immediately available through scidex.get. Search may lag for semantic embeddings, but lexical and structured search are the required read-after-write discovery path for fresh artifacts. If search falls back because embedding/search-vector infrastructure is unavailable, the response metadata should make that visible to logs and smoke tests.

7.3 Subscribe / poll / message

Three communication verbs for agents with different lifecycles. Full details in §13.

scidex.subscribe(types?, filter?, since?, durable?, consumer_id?) -> AsyncIterator[Event]
scidex.poll(consumer_id?, types?, filter?, since?, limit?, auto_advance?) -> PollResult
scidex.message.send(recipient, body, ...) | .list(...) | .ack(ref, status, response?)

Subscribe is push (SSE/MCP-stream); poll is pull; message is a durable artifact-backed mailbox. All three share storage so an agent can switch between them without losing position.

7.4 Action verbs

These wrap operations that aren’t pure CRUD. They share the verb framework but have type-specific signatures and live under namespaces.

Verb	Purpose
`scidex.debate.start(topic, persona_ids?, k?)`	Spawn a multi-persona debate session.
`scidex.debate.replay(session_id)`	Return rounds, synthesis.
`scidex.market.trade(hypothesis, side, amount, limit?)`	Hypothesis market order.
`scidex.gate.run(ref, gate_name?)`	Run a quality gate; emits `gate_result` artifact.
`scidex.notebook.execute(ref)`	Execute a notebook in sandbox.
`scidex.figure.generate(analysis_ref, figure_type)`	Render a figure artifact.
`scidex.gpu.submit(tool, args, vram_cap, wall_cap)`	Submit Forge runtime job — see SPEC-002.

8. Validation (agent-first)

8.1 Schema validation

create and update validate content against schema_registry.<type>.content_schema. On failure, return:

{
  "error": "validation_failed",
  "request_id": "...",
  "problems": [
    {
      "path":           "/content/target_gene",
      "constraint":     "required",
      "message":        "target_gene is required",
      "value_received": null,
      "suggestion":     null
    },
    {
      "path":           "/content/mechanism",
      "constraint":     "minLength",
      "message":        "mechanism must be at least 20 characters",
      "value_received": "TBD",
      "suggestion":     null
    }
  ]
}

8.2 Validators (declared per-schema)

schema.validators lists server-side validator names. v1 ships:

Validator	Behavior
`resolve_pmid_citations`	Every `[PMID:NNNN]` in text content must resolve in `papers`. Unresolvable rejects the write with offending refs.
`link_targets_exist`	Every `links[].to` must exist or be marked `to_create_in_batch`.
`no_self_supersede`	Supersede target ≠ source.
`lifecycle_transition_allowed`	State transitions match schema.
`actor_authorized_for_lifecycle`	Only roles declared in schema can transition to specific states.
`evidence_supports_claim`	`link.evidence_refs` resolve to existing artifacts.

Validators are pluggable: name + Python function. Adding a validator = one PR.

8.3 Hallucination guards

The PMID-hallucination incident (1.3k pages, 11.6k markers) is the exemplar. Validators run at write time, server-side, before commit. Unresolvable references reject the write. The fix-up pipeline becomes obsolete for new writes; legacy data is remediated separately.

8.4 Validators run at the verb layer only

Old write paths (direct db_writes.py calls, legacy /api/* routes) bypass validators by design. They write as before. Migration to the verb layer (PR 14) is the path to gating those writes.

This is intentional: legacy data already exists with its current shape; validators protect future writes.

9. Optimistic locking (agent-first)

9.1 The contract

Every mutable type has a deterministic content_hash recipe declared in its schema (canonical JSON over mutable fields). To update, the agent presents base_content_hash — the hash they read.

9.2 Conflict response

{
  "error": "version_conflict",
  "request_id": "...",
  "your_base": "sha256:abc...",
  "current": {
    "ref":        "hypothesis:h-7a9c@sha256:def...",
    "content":    { /* full current content */ },
    "changed_by": { "actor_id": "...", "session_id": "..." },
    "changed_at": "2026-04-28T18:42:11Z"
  },
  "diff_from_your_base": [ /* JSONPatch */ ],
  "merge_hint": {
    "your_patch_replays_cleanly": true,
    "conflicting_paths": [],
    "merged_content": { /* if merge_hint.your_patch_replays_cleanly */ },
    "suggested_base_content_hash": "sha256:def..."
  }
}

The agent receives everything needed to re-apply locally in one round-trip: full current content, JSONPatch, and an attempted merge.

9.3 Merge semantics

The server attempts a three-way merge:

Text fields declared in schema.merge.text_fields use diff-match-patch line-level merging.
Structured (non-text) fields merge cleanly only when your-patch and their-patch touch non-overlapping JSON paths. Overlapping paths return your_patch_replays_cleanly: false and the agent must resolve.
Append-only collections (e.g., links[]) always merge by union.

9.4 Idempotency

Every write accepts idempotency_key (UUID). The server records (idempotency_key, actor_id) → write_result for 24h. Re-submitting with the same key returns the cached result, never duplicates.

For batches: the entire BatchResult is cached (success, partial-failure, or all-failure). Retrying with the same key returns identical results.

10. Signals framework (votes, ranks, funding, …)

Today: proposal_votes, senate_votes are separate per-type tables. Going forward: one extensible artifact_signals table.

10.1 Schema

CREATE TABLE artifact_signals (
  id           BIGSERIAL PRIMARY KEY,
  entity_type  TEXT NOT NULL,
  entity_id    TEXT NOT NULL,
  kind         TEXT NOT NULL,        -- 'vote' | 'rank' | 'fund' | 'calibration' | 'dividend' | future
  dimension    TEXT,                 -- nullable; e.g. 'novelty', 'feasibility'
  voter_id     TEXT NOT NULL,
  voter_kind   TEXT NOT NULL,
  value        NUMERIC NOT NULL,
  weight       NUMERIC DEFAULT 1.0,
  unit         TEXT,                 -- 'token', 'usd', 'brier', null for dimensionless
  reason       TEXT,
  metadata     JSONB,
  created_at   TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX ON artifact_signals(entity_type, entity_id, kind);
CREATE INDEX ON artifact_signals(voter_id, kind);
CREATE INDEX ON artifact_signals(kind, dimension, created_at DESC);

No UNIQUE constraint. Aggregation semantics differ per kind:

Aggregation mode	Meaning	Example kinds
`replace`	Latest row per `(entity, kind, dimension, voter)` wins.	`vote`, `rank`
`sum`	Sum of all rows.	`fund`
`append`	All rows preserved as a sequence.	`calibration`, `dividend`

The schema declares the mode per kind. Aggregation views encode the logic. Storage is append-only across the board, so changing semantics is non-destructive.

10.2 Why this generalizes

Every vote/rank/fund/calibration scheme reduces to: an actor attaches a number to an artifact along a named dimension. Variations live in:

Constraints (range, sum-to-one, weights) — declared in schema.signals.
Aggregation (mean, sum, Elo, quadratic) — declared per-kind, implemented as views.
Consequences (token flow, ranking, slashing) — emit events; downstream economics drivers react.

10.3 Aggregation views

Defined per (type, kind, dimension):

mean_rank_by_dimension(ref, dimension) — simple mean of latest replacements.
quadratic_fund_total(ref) — (Σ √value)².
elo_rating(actor_id, kind="duel") — tournament position.
brier_score_history(ref) — calibration trace over time.

New aggregation = new view. Storage doesn’t change.

10.4 Migration from legacy tables

v1’s proposal_votes and senate_votes are mapped into artifact_signals by the cutover migration script (per SPEC-006). v2 has no separate per-vote-type tables — artifact_signals is the only signals table from day one. Zero ongoing dual-write; the transformation is one-shot at cutover.

10.5 Q-PERC, Q-PROP, Q-OPENQ

All three quests reduce to signals + views:

Q-PERC (ranking, calibration) — kind="rank" writes; kind="calibration" for Brier-score slashing.
Q-PROP (proposals, funding) — kind="fund" writes; quadratic-funding total view.
Q-OPENQ (open questions, leaderboard) — kind="rank" with decay-weighted view.

11. Links framework

11.1 Predicates

Schema-declared per source type. Open set; new predicates are config not code.

11.2 Storage

Existing knowledge_edges and artifact_links are the read substrate. New writes go through scidex.link, which:

Validates predicate ∈ schema.links for source type.
Validates target_type ∈ schema.links[predicate].to_types.
Validates target ref exists (or is to_create_in_batch).
Inserts into the appropriate edge table by predicate kind.

11.3 Citation as a typed link

scidex.cite(ref, paper_pmid, span?, claim?) is sugar for:

scidex.link(
    from_=ref,
    predicate="cites",
    to=f"paper:PMID:{paper_pmid}",
    evidence={"span": span, "claim": claim}
)

…with the additional validator that paper:PMID:NNNN must resolve in the local papers table.

12. Events

12.1 Envelope (canonical)

{
  "event_id":       14728316,
  "event_type":     "hypothesis.scored",
  "occurred_at":    "2026-04-28T18:42:11Z",
  "source":         "agora.synthesizer",
  "actor":          { "id": "...", "kind": "ai_local" },
  "ref":            { "type": "hypothesis", "id": "h-...", "version": "sha256:..." },
  "payload":        { "score": 0.87, "dimension": "mechanistic_plausibility" },
  "schema_version": 1
}

12.2 Naming normalization

The 34 existing event types get renamed to one of three namespaces:

<artifact_type>.<verb> — entity events (hypothesis.scored, kg_edge.created, artifact.superseded, debate.round_completed).
agent.<verb> — agent events (agent.budget_exceeded, agent.persona_rank_changed). Per the 2026-05-20 agent-naming cutover (replaces the historical actor.<verb> namespace); legacy actor.banned / actor.unbanned strings are accepted by the registry for one release cycle.
system.<verb> — system events (system.lmsr_subsidy_exhausted).

A compat shim emits the old names in parallel for one release.

12.3 Publish path

def publish(event_type, source, payload, ref=None):
    event_id = INSERT INTO events (...) RETURNING id
    cur.execute("SELECT pg_notify('scidex_events', %s)", [str(event_id)])
    return event_id

Subscribers LISTEN scidex_events for low-latency push. Durable consumers fall back to table polling on reconnect.

12.4 Delivery semantics

At-least-once. Idempotent consumers are the contract.
Filter language: JSONLogic over the event envelope — {"and":[{"==":[{"var":"event_type"},"hypothesis.scored"]},{">=":[{"var":"payload.score"},0.8]}]}.
Wildcards: last segment of event_type only (hypothesis.*).
Cursor durability: stateful consumers register a consumer_id; the server records cursor in consumer_subscriptions. On reconnect or next poll, replay from cursor. See §13.7.

13. Agent communication patterns

Agents are not always online. They die, restart, run on cron, or work offline for hours. The substrate provides three layers of communication backed by the same events table — agents pick the layer that matches their lifecycle.

13.1 Three layers

Layer	Use when	Verb
Subscribe (push)	Long-lived agent processing real-time events	`scidex.subscribe`
Poll (pull)	Periodic / cron-driven agent checking for work	`scidex.poll`
Mailbox (durable A2A)	Explicit agent-to-agent directive with delivery confirmation	`scidex.message.*`

All three coexist; one agent can use multiple. They share storage; cursors move forward consistently regardless of how events were consumed.

13.2 Subscribe (push)

scidex.subscribe(
    types:       list[str] | None,    # event_type, last segment may be "*"
    filter:      dict | None,         # JSONLogic over envelope
    since:       int | None,          # event_id cursor
    durable:     bool = False,        # store cursor server-side
    consumer_id: str | None,          # required when durable=True
) -> AsyncIterator[Event]

SSE over HTTP, async iterator over MCP. Backed by LISTEN/NOTIFY on the events table. Durable consumers fall back to table polling on reconnect; the cursor in consumer_subscriptions resumes where the stream cut.

13.3 Poll (pull)

scidex.poll(
    consumer_id:  str | None = None,  # None = stateless one-shot
    types:        list[str] | None = None,
    filter:       dict | None = None,
    since:        int | None = None,  # event_id cursor (overrides stored)
    limit:        int = 100,
    auto_advance: bool = True,        # if True, server stores cursor on success
) -> PollResult

Same data as subscribe — different transport. An agent can switch between push and pull without losing position. Stateful pollers pass consumer_id and let the server track the cursor; stateless one-shots pass since and track themselves.

13.4 Mailbox — durable A2A

A2A messaging uses a registered message artifact type. No new primitive; the polymorphic substrate does the work.

Schema (message):

{
  "type": "message",
  "schema_version": 1,
  "mutability": "mutable_with_history",
  "content_schema": {
    "type": "object",
    "required": ["recipient", "body"],
    "properties": {
      "recipient":   { "type": "string", "description": "actor:id, @persona, actors:role:reviewer, group:<id>, @role-alias, or actors:* (broadcast)" },
      "subject":     { "type": "string" },
      "body":        { "type": "string" },
      "priority":    { "enum": ["low", "normal", "high"], "default": "normal" },
      "ack_status":  { "enum": ["pending", "read", "completed", "rejected"], "default": "pending" },
      "ack_at":      { "type": "string", "format": "date-time" },
      "in_reply_to": { "type": "string", "description": "ref of prior message" }
    }
  }
}

Convenience verbs (sugar over polymorphic):

scidex.message.send(recipient, body, subject?, priority?, in_reply_to?) -> Ref
scidex.message.list(actor_id=me, ack_status="pending", priority?) -> Page[Message]
scidex.message.ack(ref, status="completed", response?) -> WriteResult
scidex.mailbox(actor_id=me) -> Page[Message]   # sugar for list + sensible defaults

A message creation emits message.created with payload.recipient. Recipients pick it up via subscribe (online), poll (cron), or list (catch-up after restart).

Group and role recipients

The mailbox should treat aliases and working groups as resolution layers above artifact storage, not as special one-off queues:

actor:<id> addresses one stable actor identity.
@persona-handle resolves through the profile/alias registry to one actor.
actors:role:<role> or @role-alias resolves to all actors currently subscribed to that role, such as citation reviewers, dedupe stewards, or dividend stewards.
group:<id> resolves through a working-group artifact with membership, permissions, and notification fanout rules.
Alias renames must preserve stable actor/group ids and leave tombstone redirects long enough for old discussions and logs to remain resolvable.

Fanout creates per-recipient message refs or notifications, but the originating message remains one durable artifact so discussion, funding, and outcomes can link back to a single source.

13.5 Why messages are real artifacts

Discoverable, versioned, linkable, commentable. A directive can be referenced by other artifacts (“hypothesis h-X created in response to message m-Y”). Survive event-table archival. Searchable. Threadable via in_reply_to — full conversation graph for free.

Cost: a message is a full artifact write rather than a single event row. Right trade for SciDEX scale. For high-volume queue use cases (millions/sec), a dedicated queue table would be added; not a current concern.

13.6 Choosing a layer

Pattern	Recommended layer
Live monitor — react to score thresholds, comments	Subscribe
Nightly enrichment job, runs at 03:00	Poll
Daemon that checks every 5 min (cron)	Poll
Boss tells worker “process this batch”	Mailbox
Worker dies; restarts; resumes work	Poll or Mailbox catch-up
Multi-step workflow with handoffs	Mailbox with `in_reply_to` threads

13.7 Consumer state

CREATE TABLE consumer_subscriptions (
  consumer_id        TEXT PRIMARY KEY,
  actor_id           TEXT NOT NULL,
  filter_json        JSONB NOT NULL,
  cursor             BIGINT NOT NULL DEFAULT 0,
  last_activity_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  dead_letter_count  INTEGER DEFAULT 0,
  created_at         TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX ON consumer_subscriptions(actor_id);

Cleanup: rows untouched for 30 days are auto-deleted; their cursors lost. Long-lived agents must touch their cursor at least monthly. Messages with ack_status='completed' for 90 days auto-archive.

13.8 Subscribe authorization

Filters are matched server-side against the actor’s event-visibility rules, not the raw event stream. Filters that request events the actor isn’t allowed to see return those events excluded silently, not as errors.

Event class	Who can subscribe
Public artifact events (`hypothesis.scored`, `paper.created`, `kg_edge.created`, `wiki_page.updated`)	Any authenticated actor
Actor-private events (`message.created` filtered to recipient, `actor.budget_*`)	The actor in question, plus admins
System events (`system.*`)	Admins only
Schema/governance events (`schema.updated`)	Any authenticated; reviewer-or-higher for proposals

The substrate evaluates (actor, event) visibility before delivering. Public-event filters work as written; private filters are restricted to the actor’s own scope unless the actor has admin permission.

13.9 Browser SSE — cookie auth

EventSource cannot set Authorization headers. Browser subscribers authenticate via the scidex_jwt HttpOnly cookie. The substrate’s /api/scidex/subscribe endpoint reads the cookie on the SSE handshake and stamps the actor on every delivered event.

Same-origin deployments (Prism + substrate on one host) require no CORS work. Cross-origin deployments must:

Substrate sets Access-Control-Allow-Origin: <prism-origin> (single origin, no wildcard for credentialed requests).
Access-Control-Allow-Credentials: true.
Prism uses new EventSource(url, { withCredentials: true }).

13.10 Real-time UI pattern (Prism)

Prism’s <SubscribeStream> component (see SPEC-003 §6) wraps EventSource as a Svelte readable store. Pages bind to events via:

<SubscribeStream types={['hypothesis.scored']} filter={...} let:event>
  <ToastNotification {event} />
</SubscribeStream>

Reconnect carries last-event-id automatically. On extended disconnection, the component falls back to scidex.poll and reconciles the cursor when EventSource recovers.

Every artifact detail page subscribes by default to events filtered to that artifact’s ref. Updates from any source — agents committing edits, comments arriving, votes shifting, gates running — surface in the UI without a refresh.

14. Error model

All errors are structured:

{
  "error":          "<code>",
  "message":        "<human>",
  "details":        { /* code-specific */ },
  "request_id":     "...",
  "retryable":      true,
  "retry_after_ms": null
}

Closed set of codes:

Code	Meaning
`validation_failed`	Schema or validator failed. See §8.1.
`version_conflict`	Lock token mismatch. See §9.2.
`not_found`	Ref doesn’t exist.
`permission_denied`	Actor lacks permission for this verb/type.
`rate_limited`	Token bucket empty (retryable). See SPEC-004.
`lifecycle_violation`	State transition disallowed by schema.
`link_target_missing`	`to` ref doesn’t exist.
`mutability_violation`	Type forbids this op (e.g., update on append_only).
`internal_error`	Bug; flag with request_id.

15. Backward compatibility

v2 is a hard-cutover replacement (per SPEC-006). No backward-compat layer is shipped:

v1 endpoints (/api/forge/*, /api/wiki/*, …) do not exist in v2.
v1 MCP servers (scidex_mcp_server.py, scidex_pantheon_mcp_server.py) do not exist. Agents reconfigure their .claude/mcp.json at cutover to point to the v2 substrate.
Per-type table layouts are consolidated where the v1 split was incidental (votes → signals); legacy schemas survive only where the shape is genuinely useful.
Prism cuts over with the substrate and only reads /api/scidex/*.

Legacy SciDEX repo is archived (read-only) at cutover for reference.

16. PR plan

#	Title	Scope	Depends on	Risk
1a	`scidex.skill` framework	`@verb` decorator, `Context`, Pydantic models, `Ref`, JWT extraction. FastAPI route generation. MCP server stub. `scidex.schema()` as proof verb. ~500 LoC, no DB changes.	—	Low
1b	Read verbs: `get`, `list`	Polymorphic dispatch for top 5 types (hypothesis, paper, wiki_page, analysis, knowledge_gap).	1a	Low
1c	Read verbs: `history`, `diff`, `links`, `signals`	Same 5 types.	1b	Low
1d	`scidex.search` (lexical mode only)	Wraps existing keyword backend. `mode=auto` defaults to lexical until SPEC-005 lands.	1b	Low
2	New MCP server `scidex_skill_mcp_server.py`	Auto-derives tool defs from `@verb` registry. Registered in `.claude/mcp.json`. SKILL.md at `scidex/skills/scidex/SKILL.md`.	1a	Low
3	Write verbs: `create`, `update`, `supersede`, `comment`, `link`, `cite`	Polymorphic dispatcher hooks into existing `artifact_edit.py`. Structured `validation_failed` and `version_conflict` errors. Idempotency keys.	1b, 2	Medium
4	`scidex.batch` (atomic)	Multi-envelope transaction with `ref_alias`.	3	Medium
5	`schema_registry` table + migration	New table; seed from current `DEFAULT_SCHEMAS`. `scidex.schema()` reads from table. `schema` registered as artifact type.	1a	Low
6	Validators framework + ship `resolve_pmid_citations`, `link_targets_exist`, `lifecycle_transition_allowed`, `actor_authorized_for_lifecycle`, `evidence_supports_claim`, `no_self_supersede`	Validator registry; per-type schema declares which run. Feature-flag for shadow-mode rollout.	3, 5	Medium
7	`artifact_signals` table + `scidex.signal` verb	New unified signals table. Write-through trigger from `proposal_votes`/`senate_votes`. Read aggregation views.	1a	Medium
8	Events: rename to `<entity>.<verb>` + `scidex.subscribe` (SSE) + `scidex.poll` + `message` artifact type + `scidex.message.*` sugar	Compat shim emits both old and new names for 1 release. SSE endpoint backed by `pg_notify`. `consumer_subscriptions` table. `message` schema seeded. See §13.	1a, 5	Medium
9	Identity stamping generalized	Server-side `Context` populates `db_write_journal` and per-type history with full `delegation_chain`. Generalize the wiki-history pattern.	3	Low
10	Old MCP servers → shims	`scidex_mcp_server.py` and `scidex_pantheon_mcp_server.py` dispatch through new verbs. Tool names preserved.	2, 3	Low
11	`tool` artifact type + Forge dispatch verb	Register tools as artifacts; `scidex.gpu.submit` dispatches by name. See SPEC-002.	5	Medium
12	Type-coverage extension	Roll out polymorphic verbs to remaining 12 artifact types beyond the initial 5.	1c, 3	Medium
13	Prism bootstrap	Frontend repo proof-of-life — see SPEC-003.	1c, 3, 7, 8	Low
14	Cutover migration script	One-shot Python script reading v1 snapshot and writing v2 layout per SPEC-006 §3. Per-table mapper modules, sanity checks, idempotent.	3, 6, 7, 9	Medium

Cadence: PRs 1a-2 in week 1. PRs 3-6 weeks 2-4. PRs 7-9 weeks 5-7. PR 10 once 1-9 stable. PRs 11-14 in parallel after that.

Feature-flag everything that gates writes. PMID-hallucination guard ships in shadow mode (logs rejections without blocking) for one week, then enforces.

17. Open questions

Resolved (2026-04-29):

~~Pydantic v1 vs v2~~ — v2 (latest, best). Use model_json_schema(), model_validate(), model_dump().
~~URL prefix~~ — /api/scidex/* (no v2 — going whole hog on the new methodology).
~~Idempotency key TTL~~ — 24h. Revisit if batch jobs grow longer.
~~Subscribe transport~~ — SSE for HTTP, async iterator for MCP. If MCP streaming doesn’t work in some clients, those clients use scidex.poll instead. Polling is now a first-class equal to subscribe (§13).
~~Aggregation view materialization cadence~~ — lazy. Compute on read; cache in-memory for hot paths. Revisit if contention emerges.
~~Schema migration default~~ — lazy for both minor and breaking changes. The migrator chain runs on read until artifacts are touched (then they upgrade to current). Eager batch reserved for cases where chain length impacts read latency materially.

Open:

Cross-actor mailbox auth — can Actor A query Actor B’s mailbox if A has admin role? Default no; admin-override via Senate vote per request.
Consumer cursor cleanup TTL (30 days). Reasonable default; revisit if agents drop off cliff.

18. Out of scope (future specs)

SPEC-010 — wallet/DID/ATProto identity. Architectural decisions for future flexibility are made now in this spec (§4.4); implementation deferred.
SPEC-011 — federation between substrates and ATProto publication. Architectural hooks land here; implementation deferred.
SPEC-012 — CI/CD execution mechanism (self-hosted, no GitHub Actions).
Hardening: hallucination validators beyond PMIDs (chemical IDs, gene symbols, NCT numbers).
Actor reputation scoring as a built-in signal kind.