Why it matters
Problems in training images become problems in the model. Duplicates make test scores look better than they are, broken and blurry files waste training time, and personal data in photos creates legal obligations. Cleaning a dataset is less about throwing files away and more about knowing what is in it.
1. Decide what “bad” means for your model
A blurry photo can be noise for a product-search model and useful data for a model that must handle bad cameras. Write down, in one or two sentences, what you do not want in the dataset and why. Every later step uses that sentence.
2. Check that every file opens
Decode every image, not only the file name. Look for empty files, truncated downloads, wrong file extensions and formats your training code cannot read. Keep a list of files that could not be opened instead of letting the loader skip them silently, so you know how much of the dataset you actually trained on.
3. Find exact and near duplicates
Exact copies can be found by comparing file hashes. Near copies, such as resized, recompressed or slightly cropped versions, need a perceptual comparison, for example perceptual hashes or image embeddings. Check duplicates across your training, validation and test splits: a copy that sits in both training and test inflates your results.
4. Flag blur, exposure and low-information images
- Blur: a common heuristic measures how much fine detail an image has. The right threshold depends on your data, so calibrate it on a sample you have reviewed by eye.
- Exposure: images where most pixels are clipped to black or white carry little information.
- Low information: blank frames, near-uniform colour and screenshots of error pages often slip into scraped data.
- Overlays: watermarks and burned-in text can teach a model the wrong signal.
5. Look for personal data in the photos
Faces, licence plates, screens and documents in the background are personal data under the GDPR when a person can be identified. Decide whether you need them, and whether you have a legal basis. Counting faces is often enough to understand the scale of the issue without identifying anyone.
6. Review a sample, then decide
Automated checks are a first pass. Review a sample of each flagged group before you act, and keep the originals. Move or mark files rather than deleting them, so you can change your mind when the model or the requirements change.
7. Keep a record
Write down which checks you ran, how many files each one flagged, and what you decided. A short datasheet makes the training run reproducible and is the kind of technical documentation the EU AI Act asks providers of high-risk systems to keep (Article 11 and Annex IV).