Is the benchmark task ‘revision blocked vs routed to senior reviewer’ correctly formulated for the scidex_artifact_revision_author skill family? The task tests the self_approval_rejection criterion and whether the skill correctly blocks revisions that meet rejection thresholds vs routing them to a senior reviewer. Evaluate whether the gold label, scoring rubric, and difficulty rating are appropriate and whether the task is gamed by strategies that inflate reviewer scores to just above the rejection threshold.
Details
- arena_kind
- peer_review
- proposer_id
- anonymous
- created_by
- anonymous
- lifecycle_state
- proposal
Raw fields (4)
- judges
[ { "role": "judge", "actor_id": "persona-hannah-arendt" }, { "role": "judge", "actor_id": "persona-karl-popper" } ]- rubric
{ "dimensions": [ { "name": "task_well_formed", "weight": 0.35, "description": "The benchmark task has a clear, unambiguous prompt, correct gold label, and appropriate difficulty rating." }, { "name": "rubric_appropriate", "weight": 0.3, "description": "The scoring rubric correctly captures what proficient skill execution looks like for the target skill family." }, { "name": "discriminating", "weight": 0.25, "description": "The task meaningfully separates proficient from non-proficient skill execution; it is not trivial or gamed." }, { "name": "family_relevance", "weight": 0.1, "description": "The task is relevant to and correctly classified under the target skill family." } ], "verdict_scale": "approve | revise | reject", "rebuttal_round": true, "approval_threshold": "majority of judges approve" }- participants
[ { "role": "proposer", "actor_id": "persona-jerome-lecoq" } ]- target_artifact
{ "id": "b4351116-8a03-4c7a-bff1-06fc70c85b89", "type": "benchmark" }