External Tools Landscape

What third-party tools/runtimes the substrate dispatches to and how they auth.

Source: docs/operations/external-tools-landscape.md

External Tools Landscape — operator runbook

Status: Phase A — §3.1 (schema) and §3.2 (seed importer) are shipped. §3.3 (staleness scanner) and §3.4 (benchmark linkage validator) are not yet implemented. This runbook documents what is live today and flags the pending pieces.

The external tools landscape is a curated catalog of AI-for-science tools, platforms, agents, and benchmarks — the v1 “AI Tools Landscape” — modeled as the external_tool artifact type (SPEC-083). Browse, filter, and search via standard polymorphic verbs; no bespoke /forge/landscape route logic is required on the substrate side.

System overview

external_tool artifacts (SPEC-083 §2)
  → schema: src/scidex_substrate/schema_registry/default_schemas/external_tool.json
  → seeded by: scripts/seed_external_tools.py (23 entries, SPEC-083 §3.2)
  → browsed via: scidex.list type=external_tool [filter=...]
  → searched via: scidex.search type=external_tool query=...
  → lifecycle:  draft → active → deprecated → archived

staleness scanner (SPEC-083 §3.3)        ← NOT YET SHIPPED
  → weekly job re-fetches homepage_url
  → flags entries: last_verified >90 days OR homepage 404
  → writes stale_flag signal (aggregation=replace)

benchmark linkage validator (SPEC-083 §3.4)   ← NOT YET SHIPPED
  → external_tool_benchmark_refs_resolve
  → fires on scidex.create / scidex.update for external_tool type
  → rejects benchmark_scores[] entries whose benchmark_ref does not resolve

Artifact type

external_tool is a reference catalog entry. It is distinct from the tool type (SPEC-002), which is an executable tool the substrate dispatches. An external_tool artifact describes a third-party product; the substrate catalogues it but does not run it.

Required fields

Field Type Notes
name string Display name (max 200 chars)
homepage_url URI Canonical product URL
category enum See categories below
open_source boolean Whether source code is publicly available

Optional fields

Field Type Notes
specializations string[] Free-form tags (e.g. "protein_folding")
pricing enum free / freemium / paid / enterprise / unknown
license string SPDX id or short description
github_url URI VCS repository URL
last_verified date ISO date of last successful homepage fetch
benchmark_scores[] array See §Benchmark linkage
short_description string ≤280 chars; used for catalog cards and embedding
long_description string Markdown extended description

Categories

Value Meaning
agent_platform General-purpose AI assistant or agent platform
literature Paper search, evidence synthesis, citation tools
structure Protein / macromolecular structure prediction
design De novo protein or molecule design
imaging Microscopy analysis, cell segmentation
lab_automation Liquid handling, ELN, protocol automation
benchmarking Scientific benchmarks and evaluation frameworks
data_repo Curated databases and data repositories
other Does not fit another category

Seed importer

scripts/seed_external_tools.py creates the 23 entries carried over from v1’s AI Tools Landscape. It is idempotent (duplicate creates are soft-skipped via HTTP 409).

Dry run (default)

python scripts/seed_external_tools.py

Output shows all 23 entries that would be written; nothing is changed.

Apply (write to substrate)

SCIDEX_SUBSTRATE_URL=http://127.0.0.1:8200 python scripts/seed_external_tools.py --apply

Expected output:

{
  "total": 23,
  "inserted": 23,
  "skipped": 0,
  "errors": []
}

Re-running after the first apply yields "inserted": 0, "skipped": 23 — that is correct and expected behavior.

Verify seed was applied

curl -s http://127.0.0.1:8200/api/scidex/list \
  -H 'content-type: application/json' \
  -d '{"type":"external_tool","limit":30}' \
  | python3 -m json.tool

The total field in the response must be ≥23 after one importer run.

Adding a new entry

Via HTTP (preferred for one-off additions)

SUBSTRATE_URL=http://127.0.0.1:8200
curl -s -X POST "$SUBSTRATE_URL/api/scidex/create" \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "external_tool",
    "title": "My Tool",
    "id": "extool-my-tool",
    "content": {
      "name": "My Tool",
      "homepage_url": "https://mytool.example.com",
      "category": "literature",
      "open_source": false,
      "pricing": "freemium",
      "last_verified": "2026-05-15",
      "short_description": "One-paragraph summary ≤280 chars."
    }
  }' | python3 -m json.tool

ID convention

IDs must match the pattern extool-{slug} where slug is the tool name lowercased with non-alphanumeric runs replaced by hyphens. For example: "AlphaFold 2" → extool-alphafold-2. The seed importer uses the same _slug() function; follow the same convention to avoid duplicates.

Adding entries to the seed importer

If the entry should persist across re-deployments, add it to scripts/seed_external_tools.py using _make_artifact(). Add it to the appropriate section list (LITERATURE, STRUCTURE, etc.) and ensure the ALL_ENTRIES concatenation at the bottom includes it. Update the module docstring to reflect the new totals.

Refreshing a stale entry

When last_verified is more than 90 days old, or when the homepage URL changes, update the entry via scidex.update:

SUBSTRATE_URL=http://127.0.0.1:8200
ARTIFACT_ID=extool-alphafold-2

curl -s -X POST "$SUBSTRATE_URL/api/scidex/update" \
  -H 'Content-Type: application/json' \
  -d "{
    \"type\": \"external_tool\",
    \"id\": \"$ARTIFACT_ID\",
    \"content\": {
      \"last_verified\": \"2026-05-15\"
    }
  }" | python3 -m json.tool

scidex.update bumps the version and preserves prior content per SPEC-009. Partial updates are merged; only the fields you provide are changed.

Clearing a stale_flag signal manually

Until the staleness scanner (§3.3) ships, set or clear stale_flag signals directly:

# Mark an entry as verified (clear stale_flag)
curl -s -X POST "$SUBSTRATE_URL/api/scidex/signal" \
  -H 'Content-Type: application/json' \
  -d "{
    \"type\": \"external_tool\",
    \"id\": \"$ARTIFACT_ID\",
    \"signal\": \"stale_flag\",
    \"value\": false
  }" | python3 -m json.tool

Retiring an entry

Use lifecycle transitions to deprecate an entry that has been superseded or is no longer maintained.

Deprecate (tool still exists but is superseded)

curl -s -X POST "$SUBSTRATE_URL/api/scidex/signal" \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "external_tool",
    "id": "extool-esmfold",
    "lifecycle_transition": "deprecated"
  }' | python3 -m json.tool

Then link to the successor entry:

curl -s -X POST "$SUBSTRATE_URL/api/scidex/link" \
  -H 'Content-Type: application/json' \
  -d '{
    "from_type": "external_tool",
    "from_id": "extool-esmfold",
    "link_type": "superseded_by",
    "to_type": "external_tool",
    "to_id": "extool-esm3"
  }' | python3 -m json.tool

Archive (tool is defunct)

curl -s -X POST "$SUBSTRATE_URL/api/scidex/signal" \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "external_tool",
    "id": "extool-old-tool",
    "lifecycle_transition": "archived"
  }' | python3 -m json.tool

Archived entries are excluded from browse/filter by default unless include_archived=true is passed.

Valid lifecycle transitions

From To
draft active
active deprecated
active archived
deprecated archived

Browsing and filtering

Browse uses scidex.list — no bespoke route required:

# All active entries
curl -s http://127.0.0.1:8200/api/scidex/list \
  -H 'content-type: application/json' \
  -d '{"type":"external_tool","limit":50}' \
  | python3 -m json.tool

# Filter by category
curl -s http://127.0.0.1:8200/api/scidex/list \
  -H 'content-type: application/json' \
  -d '{"type":"external_tool","filter":{"category":"structure"},"limit":20}' \
  | python3 -m json.tool

# Filter by open_source
curl -s http://127.0.0.1:8200/api/scidex/list \
  -H 'content-type: application/json' \
  -d '{"type":"external_tool","filter":{"open_source":true},"limit":50}' \
  | python3 -m json.tool

Prism renders /forge/landscape as a generic list page using these same scidex.list calls — no per-category HTML is required.

Semantic search

curl -s http://127.0.0.1:8200/api/scidex/search \
  -H 'content-type: application/json' \
  -d '{"type":"external_tool","query":"protein structure prediction","limit":10}' \
  | python3 -m json.tool

Embeddings are built from name + "\n" + short_description per the schema’s search.embedding_recipe. Ensure short_description is populated for meaningful semantic-search results.

Benchmark linkage (SPEC-083 §3.4) — NOT YET SHIPPED

Per the spec, a validator external_tool_benchmark_refs_resolve will fire on create/update and reject benchmark_scores[] entries whose benchmark_ref does not resolve to a real benchmark artifact (SPEC-023).

Current state: benchmark_scores[] entries are stored as-is without cross-reference validation. Until §3.4 ships:

  • Any string is accepted in benchmark_ref.

  • Track benchmark refs by convention: use benchmark:<id> form.

  • The links.benchmarks array (separate from benchmark_scores) can hold typed links to benchmark artifacts and is validated by the standard link resolver.

Staleness scanner (SPEC-083 §3.3) — NOT YET SHIPPED

Per the spec, a weekly scheduled job will re-fetch each entry’s homepage_url, flag entries whose last_verified is >90 days old or whose homepage returns 404, and write a stale_flag signal.

Current state: staleness detection is manual. Until §3.3 ships, operators can identify stale entries with this SQL query:

-- entries whose last_verified is more than 90 days ago
SELECT
    id,
    (content ->> 'name')           AS name,
    (content ->> 'homepage_url')   AS homepage_url,
    (content ->> 'last_verified')  AS last_verified,
    (content ->> 'category')       AS category
FROM artifacts
WHERE type = 'external_tool'
  AND lifecycle_state = 'active'
  AND (content ->> 'last_verified')::date < NOW() - INTERVAL '90 days'
ORDER BY (content ->> 'last_verified')::date;

Refresh those entries using the scidex.update flow described above.

Common operational queries

Count entries by category

SELECT
    content ->> 'category' AS category,
    COUNT(*)                AS entry_count
FROM artifacts
WHERE type = 'external_tool'
  AND lifecycle_state = 'active'
GROUP BY category
ORDER BY entry_count DESC;

Find all open-source entries

SELECT
    id,
    content ->> 'name'       AS name,
    content ->> 'category'   AS category,
    content ->> 'github_url' AS github_url
FROM artifacts
WHERE type = 'external_tool'
  AND (content ->> 'open_source')::boolean = true
ORDER BY content ->> 'category', content ->> 'name';

Find entries with stale_flag signals

SELECT
    a.id,
    a.content ->> 'name' AS name,
    s.value               AS stale_flag_value,
    s.created_at          AS flagged_at
FROM artifacts a
JOIN signals s ON s.artifact_id = a.id AND s.signal_type = 'stale_flag'
WHERE a.type = 'external_tool'
  AND (s.value::text)::boolean = true
ORDER BY s.created_at;

Look up a specific entry

curl -s http://127.0.0.1:8200/api/scidex/get \
  -H 'content-type: application/json' \
  -d '{"type":"external_tool","id":"extool-alphafold-2"}' \
  | python3 -m json.tool

Check version history of an entry

SELECT version, content_hash, created_at
FROM artifact_versions
WHERE artifact_id = 'extool-alphafold-2'
ORDER BY version DESC
LIMIT 10;

Relationship to Prism

Prism (SPEC-003) renders /forge/landscape as a generic list page over external_tool artifacts. It uses only standard scidex.list and scidex.search calls with category / specialization / pricing filters — no per-entry or per-category bespoke HTML.

If the landscape page is missing entries, verify:

  1. The seed importer ran with --apply and reported zero errors.

  2. The substrate is running and accessible at the expected URL.

  3. The lifecycle_state filter is not excluding active entries.

Access control

Initial policy (per SPEC-083 §5): any authenticated actor may scidex.create(type=external_tool). Promotion to “featured” status (when implemented) will require a Senate vote via a separate signal — the same signal/comment machinery all artifacts use.

Operators with the scidex-curator role (SPEC-020 §10) can update or retire entries without the normal comment/review cycle. Bulk operations (e.g. a category-wide last_verified refresh) should be run as the scidex-curator principal.