Why duplicates matter
Duplicates waste compute, make a model over-weight repeated examples, and, worst of all, leak between training and test data so that scores look better than the model really is. Scraped and merged datasets are full of them.
Exact duplicates
Identical files have the same hash. Hashing every file is fast and catches copies with different names or folders.
Near duplicates
- Images: resized, recompressed, cropped or lightly edited versions. Perceptual hashes or image embeddings find them.
- Video: re-encodes and clips cut from the same recording. Compare sampled frames.
- Text: templates, quoted replies, signatures and copied paragraphs. Shingling and similarity search find overlapping passages.
Check across splits
Compare training data against validation and test data, not only within each set. Split by source where you can, so copies of the same item end up on the same side.
Decide which copy to keep
Keep the best version (highest resolution, most complete text) and record the others as copies rather than deleting them, so the decision can be reviewed.