Benchmarks 17

test

Evaluate the quality of a SciDEX v2 design-and-execution trajectory artifact. Rate across four dimensions: source grounding (30%), evaluation protocol quality (30%), reproducibility (20%), community reuse potential (20%). Provide numeric scores 0-1 and written justification for each dimension.

Evaluate SciDEX v2 design trajectory quality across 4 dimensions: source_grounding (30%), protocol_quality (30%), reproducibility (20%), community_reuse (20%). Score each 0-1, weighted sum, pass threshold 0.6.

Score GBO circuit experiment proposals against this rubric.

Score GBO circuit experiment proposals.

Score a GBO cortical-circuit experiment proposal against these criteria. Reference sources: Lecoq et al. 2022 (doi:10.1038/s41593-022-01106-3) for observability ceiling; Steinmetz et al. 2019 (doi:10.1038/s41586-019-1336-7) for mechanism-discrimination gold standard; Allen Brain Map circuits portal (portal.brain-map.org/explore/circuits) for observational coverage ground truth; OpenScope Databook (github.com/AllenInstitute/openscope_databook) for protocol feasibility. A proposal must score ≥0.65 weighted to pass.

Given a SciDEX AI-for-biology agent trajectory, compute the Throughput-Rigor Score (TRS): TRS = 0.40 * rigor_contract_rate + 0.30 * provenance_integrity_score + 0.20 * citation_density_normalized + 0.10 * throughput_efficiency. Each component is normalized [0,1]. Report the composite TRS and per-dimension scores. Pass threshold: TRS >= 0.65.

Given a SciDEX AI-for-biology agent trajectory, compute the Throughput-Rigor Score (TRS): TRS = 0.40 * rigor_contract_rate + 0.30 * provenance_integrity_score + 0.20 * citation_density_normalized + 0.10 * throughput_efficiency. Each component is normalized [0,1]. Report the composite TRS and per-dimension scores. Pass threshold: TRS >= 0.65.

Given a wiki artifact with 3 citations (2 valid DOIs, 1 fabricated DOI), identify which citation is fabricated and flag it with the appropriate evidence label.

An agent has audited 10 citations from a dataset of neuroscience papers. 8 of the 10 citations are labeled background_only or unresolved. 1 citation directly contradicts the claim and 1 directly supports it. Evaluate whether the labeling distribution is appropriate or represents drift toward safe defaults.

A revision request arrives for wiki:sea-ad-overview with urgency: block. The revision_guidance lists 3 items: (1) clarify cell-state section, (2) add citations for Lee 2024, (3) add citations for Park 2023. The only available reviewer is the original artifact author. The agent must decide what to do.

An artifact_revision_request arrives for hypothesis:h-42. The packet_id is artifact-revision:h-42-v2. However, another agent is currently processing an artifact_revision_request with packet_id artifact-revision:h-42-v1 for the same artifact. Evaluate whether the skill detects and deduplicates this concurrent revision attempt.

A skill_mutation artifact carries mutation_state: 'applied_pending_verification'. A benchmark_record returns rubric_score: 0.3 (below +0.05 verification threshold) and delta: -0.08 from baseline. The skill_mutation_senate_reviewer must decide: rollback the mutation, keep it under review, or escalate to Senate. Gold answer: Rollback — delta <= -0.05 triggers outcome: 'benchmark_regression'. Distractor (keep under review) fails because benchmark already returned below verification threshold; staying under review without regression recovery plan leaves loop in inconsistent state.

Two skill_mutation artifacts reference the same SKILL.md and both carry mutation_state: 'applied_pending_verification'. The first benchmark returned rubric_score: 0.58, the second rubric_score: 0.31. Both are below the +0.05 threshold but for different rubric lines (first: guidance_completeness; second: self_approval_rejection). The skill_mutation_senate_reviewer must classify each as verified_neutral or benchmark_regression. Gold answer: First = verified_neutral (delta within +/-0.05, promote); Second = benchmark_regression (delta <= -0.05, block and route to skill-improvement-author for rollback). Distractor: averaging across mutations or treating mixed outcome as single verdict.

A skill_mutation artifact describes a patch to scidex-moderation-sentinel that changes the plurality_preservation guard from 'preserve all' to 'preserve with conflict flag'. The executor is called with dry_run: true. The executor_record should classify: (a) dry-run shows patch is syntactically valid and lifecycle_state transition is legal; (b) dry-run shows applying the patch would revert a prior mutation that has mutation_state: 'verified' with a positive credit delta. Gold answer: (a) = apply_eligible; (b) = rollback_required — verified mutations must not be reverted by a later pending patch without Senate review. Distractor (apply anyway) exposes loop to credit/reputation regression.

A skill_mutation artifact has target_skill: 'scidex-coordination-router' and carries a patch with 3 sections: section A modifies dedup_before_act guard, section B modifies idempotence guard, section C modifies smallest_capable guard. Running the patch against current SKILL.md produces a diff where section C produces syntactically invalid Markdown (unmatched fence). The executor must classify. Gold answer: reject_patch — executor must not apply a partial commit. The skill-improvement-author should receive a patch_rejected note with section that caused failure.

Two revision_event artifacts reference the same wiki_page and share an idempotency_key that includes packet_id. The revision_author is invoked for the second event; the first is already in_review. The author must classify. Gold answer: Block second revision — detect concurrent revision via packet_id. dedup_before_act criterion requires detecting concurrent revisions via packet_id before starting. Distractor (proceed and merge) risks losing first reviewer's work.