Dataset quality and AI training data
Practical checklists for teams preparing data for AI training. They describe general practice; where our own service is mentioned, it is in development.
- All file types
Dataset cleaner checklist: images, video and text for AI
Data cleaning for AI training: find broken files, duplicates, blur, frozen video and personal data in images, video and text, and record what you decided.
- Images
How to clean an image dataset before training
A practical checklist for cleaning image datasets before AI training: broken files, duplicates, blur, exposure, low-information images, personal data and a record of your decisions.
- Text
How to find personal data in text datasets before training
How to find email addresses, phone numbers, ID numbers, card and account numbers and exposed keys in text datasets before AI training, and what to do with what you find.
- Regulation
EU AI Act Article 10: what it means for your training data
What Article 10 of the EU AI Act says about training, validation and testing data for high-risk AI systems, and a practical checklist for data governance and documentation.
- Video
How to find frozen, dark and blurry footage in video datasets
How to check video datasets before AI training: frozen frames, dark and overexposed parts, blur and dirty lenses, dropped frames and broken files, reported with timecodes.
- Workflow
How to prepare data to train your own AI model
Step by step: how to prepare images, video and text to train or fine-tune your own AI model, from defining the task and collecting data to cleaning, splitting and documenting it.
- Text
How to prepare company data to fine-tune an LLM
How to turn support emails, chat logs and documents into a clean fine-tuning dataset for a large language model: format, deduplication, personal data, secrets and evaluation.
- Workflow
How to check annotation and label quality in a dataset
How to find and fix label problems in image, video and text datasets: labelling guidelines, agreement between labellers, missing and wrong labels, class balance and leakage.
- All file types
How to find duplicates and near duplicates in a dataset
Why duplicates hurt AI training data and how to find exact and near duplicates in image, video and text datasets, including across training, validation and test splits.
- Regulation
GDPR and AI training data: what to check before you train
A practical overview of GDPR questions for AI training data: when data is personal, legal basis, purpose, minimisation, anonymisation versus pseudonymisation, and documentation.
- Workflow
How to document a dataset: a practical datasheet
What to write down about an AI training dataset: purpose, sources, collection, cleaning, labelling, splits, personal data, known gaps and versions. A short template you can use.
- Video
Underwater video datasets for aquaculture: quality checks before training
How to check underwater video from fish farms before training AI models: frozen and dark footage, lens fouling, turbidity, blur and counting fish, with timecodes.
- Images
Product image datasets for e-commerce: quality checks before training
How to check product and returns photos before training AI models for search, cataloguing or damage detection: duplicates, blur, exposure, backgrounds, people and text in images.
- Regulation
EU AI Act Article 10: a training data checklist
A point-by-point checklist for Article 10 of the EU AI Act: data governance for training, validation and test data in high-risk AI systems, with the current amended text.
- Regulation
What the EU AI Act asks of every company that builds or uses AI
Not only high-risk AI: the EU AI Act duties that apply widely, such as AI literacy and transparency, and the roles of provider and deployer explained simply.
- Regulation
GPAI training data summary: what the AI Office template asks for
Article 53 of the EU AI Act: who must publish a summary of training content, what the AI Office template asks for, when to update it, and the deadlines.
- Regulation
Copyright and AI training data in the EU
What the EU AI Act and EU copyright rules ask of AI model providers: a copyright policy, respecting text and data mining opt-outs, and robots.txt.
- Regulation
Fine-tuning a model: when do you become a provider under the EU AI Act?
When modifying or fine-tuning a general-purpose AI model makes you a provider under the EU AI Act, what you must document, and the rules for high-risk systems.
- Regulation
Is my AI system high-risk under the EU AI Act?
How the EU AI Act decides whether an AI system is high-risk: products under Annex I, the eight areas in Annex III, the exception in Article 6(3) and the dates.
- Regulation
EU AI Act Annex IV: the training data section
What Annex IV of the EU AI Act asks providers of high-risk AI systems to document about training data: datasheets, provenance, selection, labelling and cleaning.
- Regulation
The EU AI Act in Norway: status in the EEA
Does the EU AI Act apply in Norway? Status of EEA incorporation, the planned Norwegian AI act, the supervisory authority, and copyright rules for training data.