What a dataset cleaner should do
Most training sets mix file types: product photos, camera footage, support emails, documents. Cleaning them means checking every file for problems that hurt the model or create legal risk, and recording what you found. A good process flags problems with a reason and leaves the decision to you, instead of silently deleting files.
Checks for every file type
- Broken files: files that do not open or stop part-way.
- Duplicates: exact and near copies, especially across training, validation and test splits.
- Personal data: anything that identifies a person, which needs a legal basis under the GDPR.
- Fit to purpose: files that do not match what the model is meant to learn.
Images
Flag blur, over- and underexposure, blank or low-information images, watermarks and burned-in text, and faces or licence plates in the background. Calibrate thresholds on a sample you have reviewed by eye. Full image checklist.
Video
Flag frozen footage, dark and overexposed parts, blur and a dirty lens, dropped frames and broken encodes, and report each problem with a start and end time so you can keep the good parts of a clip. Full video checklist.
Text
Flag email addresses, phone numbers, national ID numbers, card and account numbers, exposed API keys and tokens, copies and near copies, empty or truncated documents, and text in the wrong language or encoding. Full guide to personal data in text.
Sort, review, decide
Sort files into groups such as “fits your request”, “has what you don’t want”, “for you to check” and “could not check”. Review a sample of each group, keep the originals, and train only on what you have chosen.
Document it
Record which checks ran, how many files each one flagged and what you decided. The record makes training reproducible and supports data governance, for example under Article 10 of the EU AI Act for high-risk systems.