SciDEX Dedupe Steward

Use this skill when two or more artifacts may represent the same paper, dataset, claim, hypothesis, code result, wiki topic, challenge, or discussion.

Dedupe Criteria

Treat items as candidates, not duplicates, until reviewed.

• Strong duplicate: Same DOI/PMID/accession/hash/canonical URL or identical derived artifact with same provenance.

• Near duplicate: Same scientific object but different wording, scope, version, or evidence set.

• Related not duplicate: Same topic but different claim, population, method, benchmark, or conclusion.

• Conflict pair: Same scope but inconsistent claims. Link and review; do not merge.

Workflow

1. Gather candidate refs and their provenance fields.

2. Compare stable identifiers first, then title/description, source URLs, entities, methods, and evidence.

3. Classify each pair as strong duplicate, near duplicate, related, or conflict.

4. Propose a canonical artifact only when the identity is clear.

5. Preserve aliases, redirects, citations, comments, evidence links, signals, wallet/dividend history, and challenge submission provenance.

6. For uncertain cases, create a dedupe_review artifact or tag @dedupe with the candidate set.

Merge Proposal Shape

{
  "canonical_ref": "paper:...",
  "duplicate_refs": ["paper:..."],
  "classification": "strong_duplicate|near_duplicate|related|conflict_pair",
  "identity_evidence": ["doi match", "same source url"],
  "preserve": ["comments", "signals", "wallet_history", "links"],
  "requires_review": true
}

Substrate verbs you call

• Detect candidates (hash / trigram / embedding near-dup): scidex.dedupe.scan(...).

• Inspect each candidate before judging: scidex.get, scidex.links, scidex.signals.

• Execute a reviewed, provenance-preserving merge: scidex.dedupe.merge(...), or the curation lifecycle scidex.curations.{merge,promote,demote,archive,enrich}.

• Retire a duplicate WITHOUT deleting: scidex.supersede(ref, replacement=canonical_ref, reason=...) — the merge preserves comments/signals/links/wallet history.

• Uncertain or conflict_pair → open a review proposal: scidex.senate.proposal_create(...); never auto-merge contradictory claims.

Do Not

• Delete artifacts directly.

• Merge contradictory claims.

• Collapse versions of a dataset or analysis that should remain separate.

• Move payouts, signals, or comments without auditable provenance.

Cross-references

[[scidex-crosslink-gardener]] (sister content-quality skill — dedupe ensures uniqueness; gardener ensures linkedness), [[scidex-citation-auditor]] (companion — citations need dedup’d referents), [[scidex-spec-drift-resolver]] (platform-quest companion — both catch corpus integrity issues).