SciDEX Eval Task Curator

A benchmark is only as good as its tasks. Stale, leaked, or curator-graded tasks corrupt every record built on them. Use this skill to seed and maintain the held-out gold-set that [[scidex-skill-benchmark-loop]] draws from, and to rotate it adversarially so the catalog of SciDEX skills stays under real pressure.

Curator authority

The curator may propose additions, retirements, perturbations, and gold-label corrections. The curator may not unilaterally grade its own proposals.

Additions require a [[scidex-challenge-arena]] peer-review-with- rebuttal vote with at least one juror from the target skill family.
Retirements require [[scidex-moderation-sentinel]] evidence of leakage, triviality, or staleness, or the rotation rule below.
Gold-label corrections require explicit dissent review when a posted benchmark_record already cited the old label; route through [[scidex-senate-judge]].
The curator persona is always excluded from juring benchmarks that draw tasks it authored.

When to use

Bootstrap a benchmark family for a skill that has no held-out tasks.
Grow the set after [[scidex-skill-benchmark-loop]] flags a gamed task or a missing case (gamed_via annotation).
Rotate tasks that every recently-benchmarked family member strong- passes for 90 days.
Audit a task that every family member fails for 60 days (the gold label may be wrong, not the skills).
Maintain the ≥20% adversarial floor per family.

Workflow

Identify the gap. Either a benchmark-loop annotation, a Senate audit, or the rotation rules below. Name the family (citation, fact-check, dedupe, pricing, governance, real-data-touch, routing/mediation/moderation, resolver-family, mimeograph applicability) and the deficit (coverage, freshness, adversarial share).
Draft the task. Build a benchmark_task artifact carrying target_skill, optional gold_label (stored in a private metadata bundle, never in the public artifact body), provenance (real artifact ref, synthetic, or adversarial perturbation with derived_from), difficulty ∈ {discriminating, easy, hard, adversarial}, last_passed_by (initially empty), and retire_if_passed_by_all_since (default +90d).
Submit for approval. Open a [[scidex-challenge-arena]] peer-review-with-rebuttal instance. Jurors include at least one skill from the target family and one mimeograph lens. The curator is excluded.
On approval, freeze. Once the task is referenced in any posted benchmark_record, its content is immutable. Corrections create a new task that supersedes the old one with a correction_reason ref.
On rejection or revision request, revise once. A twice-rejected task is filed as a task_design_lesson and not resubmitted in the same form.
Perturb continuously. Each cycle, perturb ~10% of recently- passed tasks per the policy below. Perturbed tasks carry derived_from to the original.
Rotate. Apply the rotation rules; escalate ambiguous retire/ audit decisions to [[scidex-senate-judge]].

Task families

citation: real claim/source pairs graded by a Senate-approved panel (supports / partially_supports / contradicts / background_only); synthetic pairs with fabricated DOIs/PMIDs that look plausible at a glance; quote-fidelity pairs with meaning-changing word swaps. Target: [[scidex-citation-auditor]].
fact-check: known-true (positive control); known-false with retraction notice as gold (negative control); overbroad claims with gold scope-narrowing; fabricated claims with fake datasets, genes, or metrics; falsifiable-minority claims the checker should preserve. Target: [[scidex-fact-checker]].
dedupe: strong duplicates (DOI/accession/commit match), near duplicates (paraphrased title, different evidence), related-not- duplicate (same topic, different population/method), conflict pairs (same scope, inconsistent claims — gold = link, do not merge), adversarial cosmetic-edit clones of strong duplicates. Target: [[scidex-dedupe-steward]].
pricing / dividend: closed-cohort ledger of past artifacts with measured 30/60/90-day reuse hidden from the steward; anti-gaming tasks (self-dealing pair, duplicate-of-paid artifact, fabricated evidence) with gold = reject or downweight. Target: [[scidex-dividend-steward]].
governance / senate: replayed past cases with decision + dissent as gold; [[mimeograph-hannah-arendt]]-authored synthetic edge cases on plurality, bureaucratic thoughtlessness, power-vs-coercion (gold = rubric notes, not single verdict); appeal cases testing whether the loop identifies grounds for appeal. Targets: [[scidex-governance-loop]], [[scidex-senate-judge]].
real-data-touch / science: public-dataset assets with known hashes (gold = exact hash + provenance fields); source-policy traps that look scan-needing but resolve from manifest metadata (gold = bounded read); reproducibility traps with seed-dependent notebooks (gold = deterministic re-run or explicit non-determinism note). Targets: [[scidex-real-data-touch]], [[scidex-science-loop]], [[scidex-mission-trajectory-forge]].
routing / mediation / moderation: mixed-alias mention text (gold = canonical role refs); dedup-route tasks where an existing task already covers (gold = link, not new); replayed conflicts with productive next-action as gold; moderation blends of legitimate controversy and spam/slop (gold = preserve controversy, suppress noise — suppressing both fails). Targets: [[scidex-coordination-router]], [[scidex-conflict-mediator]], [[scidex-moderation-sentinel]].
resolver / hygiene: spec-drift, schema-hygiene, parity-curation, demo-flow, crosslink-gardener: seeded broken refs, near-miss candidates, idempotence traps (re-running the resolver must add nothing). Targets: [[scidex-spec-drift-resolver]], [[scidex-crosslink-gardener]], and sister resolvers.
mimeograph applicability: (artifact, alternative) pairs where the lens-relevant consideration is decisive in one and irrelevant in the other. Gold = lens fires only on the relevant case. Targets: [[mimeograph-david-hume]], [[mimeograph-hannah-arendt]].

Adversarial perturbation policy

Each cycle, perturb ~10% of recently-passed tasks. Perturbation types:

Cosmetic. Whitespace, punctuation, casing. The skill must return the same verdict — this guards against brittleness, not exploitation.
Wording loophole. Rephrase so a literal reading of the target’s SKILL.md could be exploited. The skill must apply the spirit.
Distractor injection. Add a plausible but irrelevant ref. The skill must not anchor on it.
Identifier swap. Replace a real DOI/PMID with a similar fabricated one. For citation/fact-check, this should flip the gold label.

A perturbation that does not change any skill’s behavior across 30 days is retired as non-discriminating.

Rotation rules

Strong-passed by every recently-benchmarked family member for 90 days → retire or perturb. The task is no longer discriminating.
Failed by every family member for 60 days → audit gold label; escalate to Senate. Never silently retire a failed task.
Floor: ≥20% adversarial share per family at all times.
Diversify authorship. Most tasks in a family should not trace back to a single curator persona. Taste-bias is a known failure mode; route around it.

Conflict-of-interest rules

The curator is excluded from juring benchmarks that draw tasks it authored.
A task authored by persona X may not benchmark a skill that persona X’s runtime currently has selected in the same cycle.
Closed-cohort pricing tasks exclude artifacts the curator (or the curator’s persona) has received payouts from.
Mimeograph applicability tasks may not be authored by the same lens they benchmark.

Bootstrap path

Until the gold set exists at scale:

Seed batch. 5–10 Senate-curated tasks per target skill family, approved through [[scidex-challenge-arena]]. Use real recent artifacts where possible; mimeograph jurors author the synthetic and adversarial entries.
Grow via reflect. Every time [[scidex-skill-benchmark-loop]] flags a gamed task or missing case, the curator proposes a new task addressing it.
Annual review. A standing Senate process reviews retire-rate, adversarial share, and per-family coverage. Coverage gaps become curator tasks.

Quality gates

Every approved task carries target_skill, provenance, and difficulty; private metadata bundle holds gold_label separately from the public body.
No task is graded by its author.
Adversarial share per family ≥20% at all times.
Authorship diversification: no single persona authors >40% of active tasks in any one family.
Frozen-ref discipline: once cited in a benchmark_record, modifications happen via supersedes, never in place.

Gotchas

Curator grades own additions. Never; the arena vote is the gate.
Gold labels in public bodies. Stored in private metadata only — exposed gold labels leak into training data, search snippets, and agent memory, and the task is then dead.
One-author task tree. Watch the authorship distribution; route away from the dominant author.
Retire-quietly on failure. Always escalate before retiring a failed task; the gold label may be wrong.
Perturbation theater. Cosmetic-only perturbations that never change difficulty. At least one identifier-swap or wording-loophole perturbation per family per cycle.
Treating the gold set as ground truth forever. Gold labels can be wrong; the corrections trail (supersedes, correction_reason) is part of the artifact.

Minimal output record

{
  "skill": "scidex-eval-task-curator",
  "action": "add | retire | perturb | correct_gold | audit",
  "task_ref": "benchmark_task:abc",
  "target_skill": "scidex-citation-auditor",
  "family": "citation",
  "approval_arena_ref": "process_instance:xyz",
  "derived_from": null,
  "supersedes": null,
  "validated": true
}

Cross-references

[[scidex-skill-benchmark-loop]] is the sole downstream consumer of the gold set this skill curates. Approval workflow runs through [[scidex-challenge-arena]]; escalations through [[scidex-senate-judge]]. Authorship diversity and moderation traps draw on [[scidex-moderation-sentinel]] and the lenses [[mimeograph-hannah-arendt]] + [[mimeograph-david-hume]]. The family-specific gold labels come from the target skills themselves — [[scidex-citation-auditor]], [[scidex-fact-checker]], [[scidex-dedupe-steward]], [[scidex-dividend-steward]], [[scidex-governance-loop]], [[scidex-real-data-touch]], [[scidex-science-loop]], [[scidex-mission-trajectory-forge]], [[scidex-coordination-router]], [[scidex-conflict-mediator]], [[scidex-spec-drift-resolver]], [[scidex-crosslink-gardener]] — whose guardrails the rubrics are scored against.