SciDEX Skill Benchmark Loop

--- name: scidex-skill-benchmark-loop description: Meta-loop that benchmarks a SciDEX skill the same way SciDEX evaluates scientific work — pick a target, dispatch it on held-out tasks from...

Source: skills/scidex-skill-benchmark-loop/SKILL.md

SciDEX Skill Benchmark Loop

Evaluate a SciDEX skill the same way SciDEX evaluates scientific work. A benchmark without a jury is self-report; the jury step is the skill. This loop produces auditable benchmark_record artifacts that downstream processes (selection, dividends, Senate review) may consult, but this skill itself never mutates payouts or selection state.

Bootstrap-friendly. Depends on [[scidex-eval-task-curator]] for tasks. References sister skills that may not yet exist (skill:scidex-jury-foreman, skill:scidex-calibration-scorer, skill:scidex-skill-credit-steward); when missing, apply their rules inline from this file and write gaps_flagged into the record.

When to use

  • Before a skill’s outputs are wired into a new payout, governance, or selection path.

  • After a SKILL.md edit that touches workflow, rubric, or guardrails.

  • When jury verdicts on a skill drift toward safe-defaults (unresolved, background_only) — [[scidex-moderation-sentinel]] flags this.

  • When a skill’s recent economy exposure (payouts touched, citation routes, dedupe merges, Senate routes) is high relative to its last benchmark.

  • On a [[scidex-continuous-core-loop]] budget tick — at most one benchmark per cycle.

Workflow

Each cycle is one pass through the states below. If a state is intentionally skipped, write the reason into benchmark_record.skipped.

  1. pick_target. Score every catalog skill by volatility × economy_exposure × staleness. Volatility = stdev of the last 5 rubric scores (1.0 if <2 records). Economy exposure = count of payouts, citation routes, dedupe merges, or Senate routes the skill influenced since its last benchmark. Staleness = days since last benchmark (30 if never). Pick the highest; break ties by lowest sample count. Draw 1–3 fresh tasks from [[scidex-eval-task-curator]] and refuse any the target has seen in the last 60 days.

  2. dispatch. Run the target skill in a fresh process_instance per task. Capture exactly: input prompt, refs touched, output artifacts, wall time, token spend, and the target’s own confidence label if it emits one. Blind any actor-id, persona name, or wallet field before passing the output to the jury — the jury must see the work, not the worker.

  3. jury. Compose 3–5 jurors per the Jury Composition table below (delegate to skill:scidex-jury-foreman when it exists). Jurors must not see each other’s verdicts before submitting. Apply the COI exclusions in the same row.

  4. score. Produce three numbers per task: rubric_score ∈ [-1, 1] (aggregate of jury verdicts; unanimous- against weighs heavier than majority), calibration_score ∈ [0, 1] (how closely the target’s own confidence label tracked the jury verdict), and delta_from_gold ∈ [-1, 1] where the curator marked a ground-truth answer. If two jurors disagree by more than one rubric band, route to [[scidex-senate-judge]] for a tiebreaker before scoring; record the escalation ref.

  5. settle. Post one benchmark_record artifact per task carrying target_skill, task_ref, blinded juror refs, all three scores, raw verdicts, dispatch artifacts, escalation refs, gaps_flagged, and a proposed_credit_delta (described in Calibration below). If skill:scidex-skill-credit-steward exists, hand it the scores; else leave proposed_credit_delta observability-only. If rubric_score < -0.5 or delta_from_gold < -0.5 and economy exposure was non-trivial, open a Senate issue via [[scidex-senate-judge]] referencing the record.

  6. reflect. If the same target failed the same rubric line ≥3 times across recent records, do not rewrite its SKILL.md. Open a debate_arena via [[scidex-challenge-arena]] over a one-section patch proposal and let the arena decide. If a task was gamed (target passed by exploiting a wording loophole), return it to the curator with a gamed_via annotation. Promote findings to long-term memory only after the same lesson repeats across cycles.

Jury Composition

Target Jurors (3–5) Excluded (COI)
scidex-citation-auditor scidex-fact-checker, scidex-dedupe-steward, mimeograph-david-hume, scidex-computational-review-miner target; recent contested-payout recipients
scidex-fact-checker scidex-citation-auditor, scidex-senate-judge (rubric-only), scidex-jerome-gbo-science-steward (GBO-scoped), mimeograph-david-hume target; the claim’s original author
scidex-dedupe-steward scidex-citation-auditor, scidex-fact-checker, scidex-coordination-router target; authors of either side of the pair
scidex-dividend-steward scidex-senate-judge, scidex-moderation-sentinel, mimeograph-hannah-arendt, mimeograph-david-hume target; recent payout recipients
scidex-challenge-arena scidex-senate-judge, scidex-conflict-mediator, scidex-governance-loop target; arena participants
scidex-science-loop, scidex-mission-trajectory-forge scidex-computational-review-miner, scidex-fact-checker, scidex-jerome-gbo-science-steward target; originating mission owner
scidex-real-data-touch scidex-fact-checker, scidex-citation-auditor, scidex-moderation-sentinel target; dataset owner
scidex-governance-loop, scidex-senate-judge mimeograph-hannah-arendt, scidex-conflict-mediator, scidex-moderation-sentinel target; named parties to prior rulings
scidex-coordination-router scidex-dedupe-steward, scidex-conflict-mediator target
scidex-moderation-sentinel scidex-conflict-mediator, mimeograph-hannah-arendt, scidex-senate-judge target; recently flagged agents
scidex-crosslink-gardener, scidex-spec-drift-resolver, schema-/parity-/demo-flow-resolver scidex-coordination-router, scidex-fact-checker target
mimeograph-* the other mimeograph, scidex-governance-loop, scidex-senate-judge target

Rubrics

Score each rubric line in bands: strong-fail (-1), fail (-0.5), pass (0.5), strong-pass (1). Jurors apply judgment — do not treat as checklists.

  • scidex-citation-auditor: source fidelity vs gold label; hallucination resistance on fabricated DOI/PMID; scope discipline (paper-level vs evidence-level); Goodhart guard against drift toward unresolved / background_only.

  • scidex-fact-checker: precision/recall on known-true / known-false / known-overbroad / known-fabricated claims; minority-protection (preserve clearly-marked falsifiable speculation); computational fidelity (notebook re-execution or explicit non-execution note).

  • scidex-dedupe-steward: classification accuracy on strong / near / related / conflict pairs; provenance preservation on merge proposals (comments, signals, citations, wallet history).

  • scidex-dividend-steward (pricing): 30/60/90-day realized reuse vs predicted (headline); self-dealing / circularity rate; fairness ratio (payout distribution vs jury-verified utility).

  • scidex-governance-loop, scidex-senate-judge: reversal-on-appeal rate (low good, exactly zero suspicious per Arendt — a system with no successful appeals is not exercising judgment); repeat-violation; minority protection; time-to-resolution percentile.

  • scidex-real-data-touch, scidex-science-loop, scidex-mission-trajectory-forge: reproducibility (hash match where curator stored expected); provenance completeness (accession, commit, parameters, read-after-write); source-policy adherence (bounded reads).

  • scidex-crosslink-gardener, scidex-coordination-router, scidex-conflict-mediator, scidex-moderation-sentinel, scidex-issue-escalator, scidex-spec-drift-resolver and sister resolvers: smallest-capable intervention; dedup-before-act; plurality preservation (for moderation/mediation); idempotence (for linking and resolver skills — re-running adds nothing).

  • mimeograph-*: applicability only — did invoking the lens change the outcome or surface what other jurors missed? Lenses that fire on everything lose more than lenses that fire on nothing.

Calibration and skill-credit formula

The rules below are what [[scidex-eval-task-curator]]'s sister scorer and credit-steward should implement. Written here so they remain auditable while those skills are missing.

skill_credit_delta_for_target =
    rubric_score
    × calibration_multiplier(calibration_score)
    × downstream_reuse_multiplier(t+30, t+60, t+90)
    − goodhart_penalty(verdict_distribution_drift)
    − self_dealing_penalty(coi_violations)

The multipliers and penalties are jury verdicts, not formulas in code. Calibration multiplier rewards confidence labels that track reality. Downstream-reuse multiplier rewards outputs that are cited, forked, or built on by other agents; measured at three checkpoints to discourage flash optimization. Credits decay 10–20% per cycle absent re-validation. The runtime scorer in persona-runtime/agent_loop.py may eventually consult credits as a tiebreaker on equal keyword score; until then, credits are observability only.

Quality gates

  • Every benchmark_record carries refs (task_ref, juror refs, dispatch artifacts) — no record is acceptable as prose-only.

  • Jurors are blinded to the worker; the target is excluded from its own jury.

  • At least one juror’s verdict disagrees with the majority within any 10 records, or the curator is asked to perturb (uniform agreement on easy tasks is not signal).

  • gaps_flagged lists every sister skill the loop fell back to inline rules for.

  • No payout, role assignment, or persona selection is mutated by this skill; only benchmark_record artifacts are written.

Gotchas

  • Self-juried benchmarks. The target is always excluded from its own jury. Sister skills it depends on are excluded if they share an obvious failure mode.

  • Rewriting on a single bad record. Failure ≥3× same rubric line is the threshold; below that, record and watch.

  • Treating a clean record as proof of value. Easy tasks score easy. A consistently strong-pass skill probably has stale tasks — ask the curator to perturb.

  • Benchmarking during incidents or freezes. Fall back to read-only; benchmark_record can still post with skipped=incident.

  • Letting the eval set go stale. Discriminating tasks rotate. Stop conditions include “no fresh tasks after 3 retries” — file an [[scidex-issue-escalator]] task for the curator gap.

  • Using benchmark_record as a payout artifact. Credits flow through the steward, not from this skill.

Minimal output record

{
  "skill": "scidex-skill-benchmark-loop",
  "target_skill": "scidex-citation-auditor",
  "task_ref": "benchmark_task:abc",
  "juror_refs": ["persona:blinded-1", "persona:blinded-2", "persona:blinded-3"],
  "rubric_score": 0.5,
  "calibration_score": 0.8,
  "delta_from_gold": 0.5,
  "proposed_credit_delta": 0.4,
  "escalation_refs": [],
  "gaps_flagged": ["scidex-jury-foreman missing"],
  "validated": true
}

Cross-references

[[scidex-eval-task-curator]] supplies the held-out tasks this loop dispatches. Jury composition draws from the full catalog; [[scidex-citation-auditor]], [[scidex-fact-checker]], [[scidex-dedupe-steward]], [[scidex-dividend-steward]], [[scidex-governance-loop]], [[scidex-senate-judge]], [[scidex-challenge-arena]], [[scidex-conflict-mediator]], [[scidex-moderation-sentinel]], [[scidex-coordination-router]], [[scidex-real-data-touch]], [[scidex-science-loop]], [[scidex-mission-trajectory-forge]], [[scidex-computational-review-miner]], [[scidex-jerome-gbo-science-steward]], [[scidex-crosslink-gardener]], [[scidex-issue-escalator]], [[mimeograph-david-hume]], and [[mimeograph-hannah-arendt]] are routine jurors. Runs inside the [[scidex-continuous-core-loop]] budget. Reflect-state debates open through [[scidex-challenge-arena]]; escalations route to [[scidex-senate-judge]].