Training Data Deduplication and Evaluation Contamination: Near-Duplicate Graphs, Isolation Watermarks, and Leakage Gates

Randomly splitting files 90/10 into training and evaluation sets and then deduplicating within each directory almost inevitably leaves leakage: print and mobile versions of the same web page, mirrored copies with headers, code repository forks, and paraphrases of questions and answers can land on opposite sides. A model recognizing content during evaluation does not mean it has acquired generalizable capabilities. Conversely, aggressive deduplication may also remove templates that are naturally common in the language, distorting low-resource slices.

Agent Evaluation Dataset Design discusses which tasks should be collected, while RAG Evaluation Drift discusses how production queries and offline gold standards can diverge. This article does not design tasks or retrieval metrics; it controls only membership overlap between candidate training data and retained evaluation assets.

Exact Fingerprints Come Before Near-Duplicate Decisions

Raw objects retain a byte digest, while normalized text gets a separate canonicalHash. The normalization version explicitly defines rules for Unicode, line endings, whitespace, HTML boilerplate, and code comments; it must not strip all punctuation or digits and thereby create large numbers of false duplicates. Items with exactly the same canonical hash form an exact group. The representative is selected based on source licensing, quality, time, and stable ID, not randomly according to which item happens to be scanned first.

Near-duplicate detection uses a versioned representation, such as n-gram MinHash/LSH after fixed tokenization. Code, dialogue, and image OCR require separate configurations and cannot share a single threshold. Candidate edges are verified with exact Jaccard or a domain-specific comparator before connected components are computed and assigned a duplicateClusterId. Splitting happens at the cluster level: a connected component can belong to only one of train, validation, or test. Pairwise deletion without retaining components can put A and C on opposite sides when A is close to B and B is close to C.

{
  "policy": "dedup.v5",
  "normalizer": "sha256:4a10...",
  "fingerprint": {"kind": "minhash", "ngrams": 13, "permutations": 256},
  "candidateThreshold": 0.72,
  "decisionThreshold": 0.84,
  "representativePolicy": "licensed-quality-recency-v2",
  "benchmarkFence": 3908
}

Research has shown that within-document and cross-document duplication in large language model training corpora affects memorization and evaluation; the GPT-3 work also specifically examined overlap between the training set and benchmarks. The papers provide methodological evidence, not thresholds suitable for every product. Every policy change must report samples of false removals and missed duplicates on this corpus, with particular attention to short texts, standardized legal language, code licenses, and minority languages.

Establish an Isolation Watermark for Evaluation Assets Before Data Retrieval

For every protected version, the evaluation registry stores member IDs, normalized digests, n-gram fingerprints, source timestamps, access levels, and fenceSequence. Hidden answers must not be copied into ordinary build logs; the builder receives access only to the controlled fingerprint service needed to perform overlap checks. When creating a Training Dataset Snapshot, pin the effective benchmark fence. Any candidate belonging to the same exact/near-duplicate cluster as an evaluation member enters QUARANTINED; it cannot merely be removed from test while training continues.

Advancing the benchmark registry fence, activating a snapshot, and granting training/checkpoint permits must contend on the same linearizable gate. A new evaluation version first advances the fence and blocks new runs and artifact promotion for snapshots below that watermark, then scans asynchronously for contamination; old snapshots must not continue to be consumed during the scanning window. Permits resume only after a new manifest with no overlap passes.

The scope of contamination must also expand along lineage. Paraphrases, synthetic answers, translations, or distilled samples generated from gold-standard questions must retain derivedFrom; synthetic data without provenance cannot automatically be considered clean. A public web page that legitimately entered a historical training set before an evaluation was published presents a different risk from an answer page scraped after publication. Reports therefore retain the source time, evaluation freeze time, and usage policy rather than classifying every string similarity as a violation.

If contamination is discovered only after training, evaluation conclusions for affected checkpoints should be frozen, and the overlapping clusters, the training snapshots they entered, and derived models should be listed. A few questions cannot simply be removed while the original score continues to be used; an unexposed holdout should be substituted, with retraining from a clean snapshot when necessary. Training Data Deletion Propagation uses the same reverse lineage to locate checkpoints, but contamination response aims to restore evaluation credibility and is not equivalent to deleting personal data.

Fault Injection Validates Set Boundaries

Tests change the encoding of the same content, insert headers, partially rewrite it, translate it, copy it into a fork, split it into multiple fragments, and deliberately execute the split before deduplication. They must also cover fingerprint-service timeouts, evaluation fence updates midway through a build, a normalizer implementation changing under the same label, interrupted connected-component computation, and replayed old results. An unknown check result must leave the snapshot BLOCKED; it cannot be admitted as "no match."

Gate metrics include the exact-duplicate token share, counts of near-duplicate edges and clusters, the number of cross-split clusters, exact/near benchmark overlap, unknown-provenance rate, quarantined-token share, per-language retention rate, and human-review agreement rate. Unapproved overlaps across splits or with an active benchmark must be zero; other thresholds are versioned by use case. Reports retain only necessary fingerprints and random-sample references to avoid spreading hidden evaluation text into a new contamination source.

A new policy first runs in shadow mode on the same candidates; removed members, slice distributions, and small-scale training results are compared before a new snapshot is published. Rollback may switch only to a previous snapshot that still satisfies the current benchmark fence; it cannot re-enable quarantined evaluation content simply to restore data volume. AI Supply Chain Security is responsible for proving which builder version was executed; deduplication controls must additionally prove which policy version the builder applied to the complete candidate set.

Further Reading

• Agent Evaluation Dataset Design
• RAG Evaluation Drift
• Training Dataset Snapshots
• Training Data Deletion Propagation