deidentifying-multilingual-text

De-identifying multilingual text

OpenMed de-identifies clinical text in many languages, each with a dedicated PII model, language-specific regex patterns (national IDs, phone formats), and a locale-aware surrogate generator. Pass lang= to deidentify / extract_pii and the right model, patterns, and fake-data tables are selected automatically. Everything runs on-device.

When to use this skill

Use it whenever the source text is not English, or when surrogates must look native to the locale (a German note should get German-looking fake names and a valid-format Steuer-ID surrogate, not a US SSN).

Discover supported languages at runtime — don't hardcode

import openmed
from openmed.core.pii_i18n import SUPPORTED_LANGUAGES, get_patterns_for_language

print(sorted(SUPPORTED_LANGUAGES))   # query it; the set is the source of truth
# Language-appropriate default model for a code:
models = openmed.get_pii_models_by_language("es")
# Language-specific regex patterns (national IDs, phones, etc.):
patterns = get_patterns_for_language("de")

The set currently spans English plus European, South Asian, Middle Eastern, and East Asian languages — but always read SUPPORTED_LANGUAGES rather than trusting a number, since it changes as models ship. MCP exposes the same list via openmed_list_pii_languages.

Quick start (Spanish)

import openmed

nota = (
    "El paciente Carlos Hernández (DNI 12345678Z), nacido el 11/04/1979, "
    "vive en Calle Mayor 5, Madrid. Teléfono 612 345 678."
)

result = openmed.deidentify(
    nota,
    lang="es",                # selects the Spanish PII model + ES patterns
    method="replace",         # locale-native fake values
    locale="es_ES",           # Faker locale (defaults from lang via LANG_TO_LOCALE)
)
print(result.deidentified_text)
# El paciente [surrogate name] (DNI [surrogate]), nacido el [date], ...

For German, just switch the code:

befund = "Patientin Anna Müller, geb. 11.04.1979, Steuer-ID 12 345 678 901."
result = openmed.deidentify(befund, lang="de", method="replace")

Workflow

1. Confirm the language is supported by checking SUPPORTED_LANGUAGES.
2. Pass lang= to deidentify/extract_pii. This selects the language-specific model (via get_pii_models_by_language) and the regex pattern set (via get_patterns_for_language) for national IDs and formats.
3. Set locale= for surrogates when method="replace". If omitted, the locale is derived from lang through LANG_TO_LOCALE (e.g. pt→pt_PT). Override for regional variants (pt_BR, en_GB, Gulf/Levant Arabic).
4. Let accent normalization happen. For models trained on accent-free text (Spanish), deidentify auto-strips diacritics before inference and maps spans back to the original accented text. You normally do not set normalize_accents yourself.
5. Keep surrogates stable across a document with consistent=True, seed=....

Language-specific national IDs

The pattern sets encode and the validators check real national identifier formats and checksums, so structured IDs are caught even when the model is unsure. Examples available in openmed.core.pii_i18n:

Language	Identifier	Validator
French	NIR / INSEE	validate_french_nir
German	Steuer-ID	validate_german_steuer_id
Italian	Codice Fiscale	validate_italian_codice_fiscale
Spanish	DNI / NIE	validate_spanish_dni, validate_spanish_nie
Dutch	BSN	validate_dutch_bsn
Hindi	Aadhaar	validate_aadhaar
Portuguese	CPF / CNPJ	validate_portuguese_cpf, validate_portuguese_cnpj
Turkish	TCKN	validate_turkish_tckn

These map to OpenMed CANONICAL_LABELS (ID_NUM, SSN) and are redacted by the same policy actions as any other identifier.

Hand-off to / from OpenMed

• Core de-id: deidentifying-clinical-text — methods, thresholds, keep_mapping, policies (all accept lang/locale).
• Surrogates: generating-synthetic-surrogates — locale-native fakes and custom providers per language.
• Policies: configuring-privacy-policies — policy= works with any lang.
• Audit: auditing-deidentification-runs records the model and language in the no-PHI report.
• Other surfaces: MCP openmed_deidentify / openmed_list_pii_languages; REST POST /pii/deidentify (both take a language parameter).

Edge cases & gotchas

• Never run the English model on other languages. Recall collapses. Always pass lang=; the default model is English only.
• lang ≠ locale. lang picks the detection model and patterns; locale shapes the replacement fakes. Set both when surrogate realism matters (e.g. lang="pt", locale="pt_BR").
• Some locales are approximations. Faker has no Telugu locale, so OpenMed maps te→en_IN and warns once. Override locale= if you need closer regional surrogates.
• Date order varies. Day-first languages (fr, de, it, es, nl, pt, …) parse 11/04/1979 as 11 April; the date logic is language-aware — keep lang set when shifting dates.
• Mixed-language notes (e.g. English headers in a Spanish chart) may lower recall; verify residual risk with audit=True and consider a second pass.
• No raw PHI in logs regardless of language — offsets, labels, hashes only.

Standards & references

• HIPAA de-identification, 45 CFR 164.514(b) (US) and GDPR / national DPAs for EU subjects: https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html
• National ID format references are encoded in OpenMed validators (no external registry bundled).
• OpenMed source: openmed/core/pii_i18n.py (SUPPORTED_LANGUAGES, get_patterns_for_language, validators), openmed/core/model_registry.py (get_pii_models_by_language), openmed/core/anonymizer/locales.py (LANG_TO_LOCALE).