SciDEX Skill Benchmarking

Date: 2026-05-22 Status: spec — first two skill bundles landed in PR #103; rest

Source: docs/skill-benchmarking.md

SciDEX Skill Benchmarking

Date: 2026-05-22 Status: spec — first two skill bundles landed in PR #103; rest deferred.

Goal

Make SciDEX skills first-class measurable workers. Evaluate every skill the same way SciDEX evaluates scientific work — held-out tasks, blinded juries, rubrics, calibration, downstream-reuse signals — and let the resulting benchmark records feed the value economy.

The skills already inventoried in skills/ cover the core SciDEX functions (citation auditing, fact-checking, dedupe, dividends, governance, conflict mediation, moderation, routing, real-data-touch, mission trajectories, mimeograph lenses, resolver family). What is missing is a way to ask: is each one actually doing what it says? This spec captures the design.

The motivating constraint: no hard-coded scripts. The system is prompts and loops; loops live inside SKILL.md state machines that invoke other skills. Substrate provides the data primitives (process_template, process_instance, benchmark_record, payout_schedule); the skill catalog provides the procedure.

Landed in this iteration (PR #103)

Two skill bundles, both observability-only at first — they fire on explicit invocation, do not feed selection or payouts yet:

  • skills/scidex-skill-benchmark-loop/SKILL.md — meta-loop state machine (pick_target → dispatch → jury → score → settle → reflect) with per-target jury composition, per-family rubrics, calibration + skill- credit formula written out for auditability.

  • skills/scidex-eval-task-curator/SKILL.md — held-out gold-set curator with COI rules (no self-grading), adversarial perturbation policy (≥20% adversarial floor, ~10%/cycle perturbation), rotation rules, bootstrap path through scidex-challenge-arena peer-review-with-rebuttal approval.

Both are listed in _PRIOR_OR_UTILITY_EXCEPTIONS in tests/test_skills_personas_consistency.py with a written rationale.

Architecture

Two ledgers, not one

  • Persona dividend ledger (existing skill:scidex-dividend-steward): rewards contributions to the contribution — durable scientific utility. Unchanged by this spec.

  • Skill credit ledger (new, sibling steward): per-skill balance that decays 10–20% per cycle absent re-validation. The runtime scorer in persona-runtime/agent_loop.py may eventually consult credits as a tiebreaker on equal keyword score; near-zero-credit skills demoted to read-only or quarantined pending Senate review.

The credit formula (descriptive, not computed)

skill_credit_delta =
    rubric_score
    × calibration_multiplier(calibration_score)
    × downstream_reuse_multiplier(t+30, t+60, t+90)
    − goodhart_penalty(verdict_distribution_drift)
    − self_dealing_penalty(coi_violations)

Multipliers and penalties are jury verdicts, not numbers a script computes. Calibration rewards confidence labels that track reality; downstream-reuse rewards outputs that are cited, forked, or built on by other agents; three checkpoints (30/60/90 days) discourage flash optimization.

Per-family rubric coverage

The benchmark loop ships with rubric bands for every operational skill family (skill:scidex-skill-benchmark-loop § Rubrics). Each band is strong-fail (-1) | fail (-0.5) | pass (0.5) | strong-pass (1). Jurors apply judgment, not checklists.

Coverage:

  • citation-auditor: source fidelity, hallucination resistance, scope discipline, Goodhart guard.

  • fact-checker: precision/recall on labeled gold, minority protection, computational fidelity.

  • dedupe-steward: classification accuracy, provenance preservation.

  • dividend-steward (pricing): 30/60/90-day realized reuse, self-dealing / circularity, fairness ratio.

  • governance-loop / senate-judge: reversal-on-appeal (low good, zero suspicious per Arendt), repeat-violation, minority protection, time-to-resolution.

  • real-data-touch / science-loop / mission-trajectory-forge: reproducibility hash match, provenance completeness, source-policy adherence.

  • coordination-router / conflict-mediator / moderation-sentinel / issue-escalator / crosslink-gardener / resolver family: smallest-capable intervention, dedup-before-act, plurality preservation, idempotence.

  • mimeograph-*: applicability only — did the lens change the outcome? Lenses that fire on everything lose more than lenses that fire on nothing.

Quality-of-comments/debates measurement

Not in PR #103. Three signals, all judged:

  1. Conviction shift — participants log a 0–100 confidence on the contested proposition before and after; quality = magnitude of shift × evidence-grounded-ness of the shift (jury reads the comment and asks: new evidence, or social pressure?).

  2. Downstream materialization — did the comment lead to an artifact edit, evidence link, correction, new claim, or follow-up experiment within N days? Pure thread-length is ignored.

  3. Survives moderation lens — sentinel and Arendt mimeograph audit the comment for noise / restatement / role-tag abuse.

Debate quality aggregates (1) over participants, (2) over artifacts produced, plus a convergence-vs-fragmentation classifier (jury labels each round as crux-narrowing or crux-multiplying).

Value-gap / experimental-proposal measurement

Not in PR #103. Mechanism is the prediction-market review already enumerated in skill:scidex-challenge-arena:

  • Author labels expected outcome (effect size, success probability, cost).

  • skill:scidex-mission-trajectory-forge opens a prediction market over the proposal’s resolvable claim, fixed settlement date.

  • At settlement: oracle = experimental result or replication review.

  • Score = Brier(author_forecast) and Brier(market_price_at_close).

  • Credit flows to the proposer scaled by realized utility minus cost; market makers and forecasters get a sliver.

Value-gap metric = (median predicted utility of accepted proposals over a window) − (median realized utility 90 days later). Persistently positive ⇒ system pays for vapor. Persistently negative ⇒ system under-prices real work / under-funds.

Anti-gaming

  • Sentinel watches the benchmark itself — sudden drift toward safe-but- empty outputs flags as a rubric line, not as platform noise.

  • Curator rotates 10% of recently-passed tasks adversarially per cycle; cosmetic, wording-loophole, distractor-injection, and identifier-swap perturbations.

  • Jury composition is randomized + COI-filtered; the target never juries itself.

  • Senate has reset authority — if a rubric is wrong, Senate can invalidate a benchmark window through the same appeal path that applies to sanctions.

Open items

See docs/tasks/skill-benchmarking-task.md for the work breakdown. In priority order: sister skill bundles (jury-foreman, calibration-scorer, skill-credit-steward), seed batch through Senate-curated arena vote, persona pinning, runtime scorer wiring, comment/debate measurement, prediction-market value-gap loop.

Cross-references

  • Landed skills: skills/scidex-skill-benchmark-loop/SKILL.md, skills/scidex-eval-task-curator/SKILL.md.

  • Skill catalog: skills/README.md under Meta-evaluation.

  • Maintenance ownership conventions: skills/MAINTENANCE-COVERAGE.md.

  • Dependent skills: scidex-challenge-arena (peer-review-with-rebuttal for seed batch + reflect-state patches), scidex-senate-judge (escalation), scidex-dividend-steward (sibling to the future skill-credit-steward), scidex-moderation-sentinel (benchmark drift detection), scidex-continuous-core-loop (budget envelope).

  • Substrate primitives this design depends on: process_template / process_instance, benchmark_record (new artifact type proposed), payout_schedule.