Fine-tuning or retrieval?
Fine-tuning changes how a model behaves: its tone, format and the tasks it handles well. If you mainly want it to answer from your documents, retrieval (searching your documents at question time) is often simpler and easier to keep up to date. Many teams use both. Either way, the data has to be clean.
1. Choose examples that show the behaviour you want
A fine-tuning set is usually pairs of input and ideal output, for example a customer question and the best answer your team gave. A smaller set of good, consistent examples tends to beat a large set of mixed quality.
2. Find copies and near copies
Support data is full of templates, signatures and repeated threads. Duplicates make the model repeat boilerplate and can leak between training and evaluation data. Find and mark duplicates before you split.
3. Find personal data and secrets
Emails and chats contain names, email addresses, phone numbers, ID numbers, card and account numbers, and sometimes passwords or API keys. A model can reproduce what it was trained on. Find them, then mask, remove or keep them only with a legal basis. Revoke any exposed keys. See personal data in text datasets.
4. Check language, encoding and length
Flag empty or truncated messages, broken characters, auto-replies and text in languages you do not want the model to answer in.
5. Keep a held-out evaluation set
Set aside real examples the model never sees in training and judge it on those, ideally with the people who answer customers today.
6. Record what went in
Note sources, date ranges, filters, what personal data was found and how it was handled. You will need it when someone asks what the model was trained on.