The complete guide to finding personally identifiable information in unstructured data: what counts as PII, how AI-based detection actually works, which entity types matter, and how to choose or build a detection system.
Start ReadingPII detection is the automated process of finding, classifying, and locating personally identifiable information inside data — especially unstructured data such as free text, documents, chat transcripts, emails, logs, images, and audio transcriptions. A detection system takes raw content as input and returns a structured answer: which pieces of personal information are present, of which type, at which exact position, and with what confidence.
The "locating" part is what separates detection from mere classification. Knowing that a document "contains PII" is nearly useless operationally; knowing that characters 118–129 are a phone number and characters 205–216 are a Social Security number is what makes every downstream action possible — redaction, masking, alerting, quarantining, indexing for deletion requests, or blocking a message before it reaches a large language model. That is why PII Detection API returns each entity with its type, matched text, character offsets, and confidence score, plus an optionally masked version of the input in the same response.
Detection matters because personal data does not stay where it was designed to live. Databases have schemas that tell you the email column holds emails, but the same email address also appears in a support ticket, a log line, a screenshot, a call transcript, and a CSV attachment. Industry studies consistently find that the majority of an organization's sensitive data lives in this unstructured long tail, invisible to schema-based governance. PII detection is the instrument that makes that invisible data visible — and therefore governable.
A useful mental model: PII detection is to privacy engineering what a vulnerability scanner is to security engineering. It does not by itself make you compliant, but nothing else in the compliance program works without the inventory it produces. You cannot minimize, delete, disclose, or protect data you have not found.
These four terms overlap but are not interchangeable — each comes from a different legal regime, and the differences drive which entity types you must detect and how you must handle them.
The broadest term: any information that can identify a specific individual, alone or in combination. US usage (NIST SP 800-122) distinguishes information that identifies directly from information that is "linkable"; the EU's GDPR uses the even wider term personal data — anything relating to an identified or identifiable person. PII spans names, contact details, government IDs, and online identifiers such as IPs and cookies.
A US legal category created by HIPAA: individually identifiable health information held or transmitted by a covered entity or its business associates. PHI is PII plus health context — a name in a hospital's billing system is PHI; the same name in a retailer's CRM is not. HIPAA defines a concrete list of 18 identifiers whose removal de-identifies a dataset under the Safe Harbor method.
Cardholder data as defined by the PCI DSS contractual standard: the primary account number (PAN) above all, plus cardholder name, expiration date, and service code, with stricter rules for sensitive authentication data (CVV, track data, PINs) which may never be stored after authorization. PCI scope is binary and unforgiving — one stored PAN in a log pulls the whole system into audit scope.
A subset of personal data given heightened protection because misuse causes disproportionate harm. GDPR Article 9 lists racial or ethnic origin, political opinions, religious beliefs, trade-union membership, genetic and biometric data, health data, and data about sex life or sexual orientation — processable only under narrow exceptions. Detection systems flag these as distinct entity types precisely because the legal bar for handling them is higher.
ETHNIC_GROUP, RELIGION, POLITICAL_AFFILIATION, SEXUAL_ORIENTATION, BIOMETRIC_DATANot all PII identifies equally, and the distinction shapes every detection policy. A direct identifier points at one person on its own: a Social Security number, a passport number, an email address, a full name in a small population, a medical record number. If a direct identifier leaks, the person is identified — no cleverness required.
A quasi-identifier (or indirect identifier) does not identify anyone alone but becomes identifying in combination: birth date, gender, ZIP code, job title, employer, city, age, ethnicity. The canonical result, from Latanya Sweeney's work, is that roughly 87% of the US population is uniquely identified by just three quasi-identifiers — 5-digit ZIP code, birth date, and gender. This is why "we removed the names" has never been an anonymization strategy: the quasi-identifiers left behind re-identify most of the dataset when joined against a voter roll or a data-broker file.
For detection engineering, the consequence is that a serious detector must cover both classes. Direct identifiers are the easier half — many have validatable structure (checksums in card numbers and IMEIs, format rules in passport and driver's license numbers). Quasi-identifiers are harder: "34-year-old teacher from Leeds" contains three of them and not a single regex-matchable token. Only a model that understands language can flag AGE, EMPLOYMENT, and CITY in that sentence.
Policy then decides what to do with each class: mask direct identifiers always; generalize or evaluate quasi-identifiers based on the dataset's purpose and re-identification risk. The API supports this split natively — request different entity lists for different pipelines, or use exclude_entities to keep analytically valuable quasi-identifiers while stripping everything direct.
Modern detection stacks four complementary layers. Each catches what the previous one misses, and together they explain why transformer-era detection decisively outperforms the regex scanners of the 2010s.
The core is named entity recognition: a transformer network reads the full text and tags each token with an entity label, using bidirectional context. This is how "Paris" gets tagged as a person in "Paris signed the contract" but not in "flights to Paris", and how an unlabeled 7-digit number in a chart header is recognized as a medical record number. NER is what finds entities that have no fixed format at all — names, addresses, diagnoses, employers. See NER vs regex compared.
Many identifiers carry mathematical structure: credit card numbers satisfy the Luhn checksum, IBANs pass a mod-97 check, IMEIs have a check digit, SSNs exclude specific ranges, IPv4 octets stay under 256. A validator layer confirms or vetoes candidate matches, collapsing the false-positive rate for numeric entities. A random 16-digit number passes Luhn about 10% of the time — validation plus context turns "maybe" into a reliable score.
The words around a candidate change everything. "Card ending 4242", "MRN:", "connected from", "DOB" are strong positive signals; "version", "order #", "invoice" are negative ones. Context analysis is also how the system distinguishes a 9-digit SSN from a 9-digit routing number, and how custom_instruction works — natural-language rules like "ignore employee IDs starting with EMP" are applied as contextual overrides without retraining anything.
Every match emerges with a probability from 0 to 1, fusing model certainty, validator results, and context strength. The threshold parameter turns this into a policy dial: healthcare pipelines run low thresholds because a missed identifier is a breach, analytics pipelines run higher ones because over-masking destroys utility. Scores also enable human-in-the-loop routing — auto-handle the confident matches, queue the borderline ones. See measuring detection accuracy.
A detector is only as useful as its taxonomy. PII Detection API organizes 150+ entity types into families that map cleanly onto regulatory categories; the table shows the families with representative types and the dedicated guide for each. The complete list lives on the entities page.
| Family | Representative Entity Types | Detection Guide |
|---|---|---|
| Identity | PERSON_NAME, DATE_OF_BIRTH, AGE, GENDER |
Detect names · Detect dates of birth |
| Contact | EMAIL_ADDRESS, PHONE_NUMBER, ADDRESS, ZIP_CODE, GPS_COORDINATES |
Detect emails · Detect phones · Detect addresses |
| Government IDs | SSN, NATIONAL_ID, TAX_ID, PASSPORT_NUMBER, DRIVERS_LICENSE_NUMBER |
Detect SSNs · Detect passports · Detect licenses |
| Financial | CREDIT_CARD_NUMBER, CVV_NUMBER, IBAN_CODE, SWIFT_BIC, ROUTING_NUMBER, FINANCIAL_ACCOUNT_NUMBER |
Detect card numbers · Detect bank accounts |
| Network & Device | IP_ADDRESS, MAC_ADDRESS, DEVICE_ID, IMEI, COOKIE, USER_AGENT |
Detect IPs & device IDs |
| Health (PHI) | MEDICAL_RECORD_NUMBER, HEALTH_INSURANCE_ID, DIAGNOSIS, PRESCRIPTION, BLOOD_TYPE |
Detect MRNs & PHI |
| Credentials & Secrets | API_KEY, PASSWORD, AUTH_TOKEN, SSH_KEY, AWS_CREDENTIALS, DATABASE_CONNECTION_STRING |
Scanning logs for PII & secrets |
| Special Category | ETHNIC_GROUP, RELIGION, POLITICAL_AFFILIATION, SEXUAL_ORIENTATION, BIOMETRIC_DATA |
GDPR guide (Art. 9) |
| Employment & Misc | EMPLOYMENT, MARITAL_STATUS, SERIAL_NUMBER, URL |
Full entity reference |
The same detect-classify-locate primitive powers very different systems. These six patterns cover most production deployments — each links to a dedicated implementation guide.
Scan prompts before they reach a model and responses before they reach users, preventing personal data from entering third-party model providers, training corpora, or vector stores. The fastest-growing deployment pattern. Guides: LLM guardrails, chatbot filtering, RAG pipelines.
Inspect content at egress points — outbound email, file uploads, API responses, ticket exports — and block, mask, or alert when sensitive entities are found. Context-aware detection cuts the false-positive noise that made first-generation DLP unusable. Guides: DLP, email scanning.
Sweep databases, data lakes, and document stores to build the personal-data inventory that GDPR Art. 30 records, DSAR fulfillment, and retention enforcement all depend on. Free-text columns and BLOBs are where schema-based tools go blind. Guides: database discovery, document scanning.
Sanitize telemetry at ingest so identifiers never reach long-term storage: a masking step in Logstash, Kafka, Spark, or Airflow keeps observability data useful while stripping IPs, device IDs, emails, and stray secrets. Guides: log scanning, ETL & streaming pipelines.
Customers volunteer card numbers, SSNs, and health details in tickets and chats regardless of policy. Real-time masking before storage keeps helpdesk archives, QA reviews, and training datasets clean. Guide: support ticket PII detection; industry view: call centers.
Produce privacy-safe copies of production data for development, QA, analytics, and research — masking direct identifiers while preserving structure and utility, with consistent hashing to keep referential integrity across tables and documents. Guide: test data PII detection.
Every major privacy regime implicitly assumes you can find personal data on demand — for access requests, breach reports, deletion, and minimization. Detection is the shared technical substrate underneath all of them.
| Regulation | Scope | What Detection Enables | Guide |
|---|---|---|---|
| GDPR (EU/EEA) | All personal data of people in the EU, processed by controllers/processors anywhere | Art. 30 data mapping, DSAR search, Art. 17 erasure, breach-scope assessment, minimization, pseudonymization as an Art. 32 measure | GDPR PII detection |
| HIPAA (US) | PHI held by covered entities and business associates | Safe Harbor de-identification (the 18 identifiers), minimum-necessary enforcement, breach risk assessment | HIPAA PHI detection |
| CCPA/CPRA (California) | Personal information of California residents, incl. households and probabilistic identifiers | Consumer access/deletion requests, sensitive-PI handling, data-sharing inventories | CCPA/CPRA guide |
| PCI DSS (contractual) | Cardholder data wherever cards are accepted or processed | Req. 3 storage minimization, PAN discovery in logs and files, scope reduction evidence | PCI DSS discovery |
| FERPA, GLBA, state laws… | Education records, financial customer data, and a growing patchwork of US state privacy acts | The same primitives: inventory, subject-request search, redaction before disclosure | Education · Banking |
Every engineering team's first instinct is "we could write some regexes for this in a sprint." Some genuinely should. Most discover that the sprint becomes a permanent team. Here is the honest calculus.
A homegrown detector is defensible when your problem is genuinely narrow and static: one or two entity types with rigid formats (your own account-number scheme), one language, one data source, and tolerance for imperfect recall. A validated regex plus a checksum can be excellent at exactly that. It also makes sense when you have an NLP team whose core product is detection and the investment compounds.
The visible cost is writing patterns; the real cost is everything after. Names, addresses, and diagnoses have no format — they require trained NER models, which require labeled data, evaluation sets, and retraining as your text drifts. Then multiply by languages (60+ supported here), by entity types (150+), and by the false-positive triage burden your security team inherits. Detection quality is also adversarial to measure: you don't know what you missed until an audit or a breach finds it for you.
An API converts the problem from research to integration: one POST request with the documented contract, entity selection per pipeline, thresholds and natural-language exclusions instead of model surgery, and accuracy that is continuously maintained against new formats and languages. Deployment stays flexible — cloud API for most workloads, on-premise when data cannot leave your boundary. Costs scale with usage (pricing) rather than with headcount, and you can validate quality on your own data in the demo before committing anything.
Whether you build or buy, evaluate the same way. First, measure on your own data, not on vendor demos: label a few hundred representative samples — your tickets, your logs, your notes — and compute precision, recall, and F1 per entity type. Aggregate scores hide exactly the failures that matter; a detector can post 95% overall while missing half of your driver's license numbers. Our accuracy guide gives a complete methodology and a labeling protocol.
Second, decide which error is expensive for you. Compliance and de-identification pipelines pay for false negatives (missed PII becomes a breach), so they favor recall and run low thresholds. Analytics and search pipelines pay for false positives (over-masking destroys utility), so they favor precision. A good detector exposes this trade-off as a dial — confidence scores and a threshold parameter — rather than hard-coding one policy.
Third, check the operational envelope: latency at your payload sizes (real-time chat needs sub-second; batch archives don't), throughput and rate limits, language coverage, offset fidelity (character positions must survive round-trips for downstream redaction), customization without retraining, deployment options, and the security posture of the vendor itself — a PII detector sees your most sensitive data, so GDPR, retention policy, and an on-premise escape hatch are not nice-to-haves.
Finally, test the edge cases that break naive systems: identifiers split across lines, mixed languages in one message, look-alike negatives (version numbers vs IPs, order IDs vs SSNs), and text from OCR with character-level noise. Five minutes pasting adversarial samples into the live demo tells you more than any datasheet.
One endpoint, one JSON body, structured entities back. The same request shape works from any language — here it is in cURL, Python, and JavaScript.
# cURL — detect all entity types with default settings
curl -X POST https://piidetectionapi.com/api/moderate.php \
-H "Content-Type: application/json" \
-d '{
"api_key": "YOUR_API_KEY",
"api_type": "pii_detection",
"text": "Contact John Doe at [email protected] or 555-123-4567.",
"mask_mode": "replace"
}'
# Python — scoped detection with a confidence threshold
import requests
resp = requests.post(
"https://piidetectionapi.com/api/moderate.php",
json={
"api_key": "YOUR_API_KEY",
"api_type": "pii_detection",
"text": "Contact John Doe at [email protected] or 555-123-4567.",
"entities": ["PERSON_NAME", "EMAIL_ADDRESS", "PHONE_NUMBER"],
"threshold": 0.6,
"mask_mode": "replace",
},
timeout=30,
)
data = resp.json()
for e in data["detected_entities"]:
print(e["type"], e["text"], e["start"], e["end"], e["confidence"])
print(data["anonymized_text"])
# Contact [NAME] at [EMAIL] or [PHONE] .
// JavaScript (Node fetch) — consistent hashing for analytics-safe output
const resp = await fetch("https://piidetectionapi.com/api/moderate.php", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
api_key: process.env.PII_API_KEY,
api_type: "pii_detection",
text: userMessage,
exclude_entities: ["URL"], // keep links intact
mask_mode: "hash", // same input -> same token
custom_instruction: "Do not flag our internal ticket IDs like TCK-12345",
}),
});
const { detected_entities, anonymized_text } = await resp.json();
Full parameter reference, response schema, and language SDKs are in the API documentation; the request/response contract is summarized on the API overview.
anonymized_text in the chosen mask mode.Paste a sample into the live demo and watch 150+ entity types light up with offsets and confidence scores — then pick a plan when you are ready to integrate.
Try the Live Demo View Pricing