· 2 min read · NETBRAND CORP
How to run a de-identification POC: 100 to 200 real documents and recall per entity
A practical plan to evaluate PII redaction tools on your own clinical documents: sampling, annotation, metrics per entity type and the acceptance criterion your committee needs.
Why your own documents
Public benchmarks use other hospitals, other templates and often synthetic notes. Your EHR exports have their own headers, abbreviations and identifier formats. The only number that predicts production behaviour is the one measured on your documents.
Step 1: sample
- Take documents from the sources that produce the most volume: discharge summaries, progress notes, referrals, scanned forms.
- Mix native and scanned PDFs in the proportion you actually receive.
- 100 to 200 documents is usually enough to see where a tool fails; add more for rare document types.
Step 2: annotate
Two people annotate independently, then resolve disagreements. For each span record the entity type: name, SSN, MRN, member ID, address, ZIP, date, phone, email, facility, and any custom type you care about. Keep the annotation guideline short and written down.
Step 3: run the candidates
Run every tool on the same set, on-premise, with the same configuration you would use in production. Save the raw detections, not just the redacted output, so you can score them.
Step 4: score per entity type
- Recall: of the identifiers a human marked, how many did the tool find? This is the privacy metric.
- Precision: of what the tool marked, how much was really an identifier? This is the utility metric; over-redaction destroys clinical meaning.
- Report both per entity type. A single global score hides the failures that matter.
Step 5: agree the acceptance criterion
Write it down before you see results. For example: recall of 99% or higher on direct identifiers, measured on the validation set, plus monthly human sampling in production. The legal framing is in Pseudonymization vs. anonymization under HIPAA, GDPR, LGPD and Colombia's Law 1581.
Step 6: decide the policy per entity
Once you trust detection, choose what happens to each type: tokenize names, shift dates, generalize ages and ZIP codes, redact SSNs. That policy, the metrics and the sampling plan are the document your committee signs.
Run it with Velo
A Velo POC follows exactly this plan, on-site, in one to three weeks, and ends with recall per entity type and a report for your committee. For the architecture that the POC protects, see How to send clinical notes to a cloud LLM without sending the patient.
This article is general information, not legal advice.