Why it matters
Support emails, chat logs, documents and scraped web text often contain personal data. Under the GDPR, personal data in a training set needs a legal basis and a clear purpose, and a model can repeat what it has seen. Finding it before training is much cheaper than dealing with it afterwards.
What to look for
- Contact details: email addresses, phone numbers and postal addresses.
- National ID numbers: for example the Norwegian national identity number (11 digits; under the current scheme the last two are check digits).
- Financial numbers: card numbers (which pass the Luhn check) and bank account numbers such as IBANs (which pass a mod-97 check).
- Names of private persons, especially next to other details.
- Secrets: API keys, access tokens and passwords pasted into tickets or code.
How to find it
- Patterns with validation. Regular expressions find candidates; check digits remove many false matches. About one in ten random 16-digit numbers passes the Luhn check, so combine it with known card prefixes and context.
- Named-entity recognition finds names and places that no pattern can describe. It is useful, but it misses some names and flags some ordinary words, so treat it as a pointer, not a verdict.
- Secret scanning looks for the known formats of keys and tokens, and for long random strings in places where secrets usually appear.
- Context. “Call me on” before a number, or a signature block at the end of an email, raises the confidence of a match.
What to do with what you find
For each type, decide whether the model needs it. Common choices are to leave the document out, to mask the value (for example replacing an email address with a placeholder), or to keep it where you have a legal basis and a reason. Note that pseudonymised data, where a person can still be identified with additional information, is still personal data under the GDPR. Exposed keys should be revoked, not only removed from the dataset.
Expect misses and false alarms
No scanner finds everything. Review a sample of flagged and unflagged documents, measure how often the checks are wrong on your own data, and keep a record of what you scanned, what you found and what you decided.