unicode-consistency

Unicode Consistency

Audit and remediate Unicode issues in text columns.

When to invoke

• User suspects "weird characters" in the data.
• Search / joins on text keys return inconsistent results (usually a normalisation or invisible-char issue).
• Dataset came from multiple sources with different encodings / input methods.
• Preparing text for embedding (see vector-upsert) or search indexing.

Checks performed

1. File / column encoding — detect declared vs actual encoding using chardet or charset-normalizer; flag mismatches.
2. Normalisation form — for each text column, sample values and check whether they are consistently NFC (or NFD/NFKC/NFKD). Mixed forms cause key mismatches that look identical to a human.
3. Script mixing within a value — e.g. Latin + Cyrillic in the same "word" (аpple with Cyrillic а instead of Latin a). Detect via unicodedata.category / unicodedata.name script inspection.
4. Confusables / homoglyphs — use the Unicode confusables table (confusable_homoglyphs / uniseg) to flag values likely to be spoofed or accidentally copy-pasted.
5. Invisible characters — zero-width space (U+200B), zero-width joiner (U+200D), soft hyphen (U+00AD), left-to-right / right-to-left marks (U+200E/F), BOM (U+FEFF) embedded mid-string.
6. Whitespace variants — non-breaking space (U+00A0), em/en spaces, thin space, ideographic space — all commonly confused with regular space.
7. Quote / dash variants — smart quotes (", ', ') vs ASCII quotes, em-dash / en-dash vs hyphen-minus.
8. Emoji and sequence completeness — broken zero-width-joiner sequences from lossy copy/paste.
9. Mojibake patterns — double-encoded UTF-8 (é for é, ’ for ', 中 for 中). Detect via statistical patterns.
10. Case-folding pitfalls — Turkish I/ı, German ß — flag if the dataset relies on case-insensitive matching.
11. Combining marks not attached — orphan combining characters.
12. RTL / bidi isolation — mixed LTR/RTL text without proper bidi markers for display.

Procedure

1. Profile text columns — list columns with dtype == object and sample values.
2. Run the checks above per column; aggregate counts of each issue category.
3. Produce a report unicode_report.md:
   • Per column: issues detected, counts, sample rows (masked if PII), severity.
   • Severity rubric: mismatched normalisation = high (silent joins break), mojibake = high (lossy), confusables = medium (often benign but can indicate spoofing), invisible chars = medium, cosmetic whitespace = low.
4. Propose a remediation script per column with:
   • unicodedata.normalize('NFC', s) (or the user's preferred form).
   • Strip invisible characters (whitelist of explicitly-allowed ones, remove the rest).
   • Replace non-breaking spaces with regular spaces (if appropriate).
   • Normalise quotes/dashes if the domain wants ASCII.
   • Fix known mojibake via ftfy.fix_text(...).
5. Preview on sample before applying — show 10 before/after pairs and confirm with user.
6. Apply to a new column (<col>_nfc or <col>_clean) by default; overwrite only on explicit request — and follow the backup policy in CONVENTIONS.md.
7. Re-run the audit to confirm residual issues.
8. Update the data dictionary — record the normalisation form and cleaning rules applied per column.

Dependencies

pip install pandas charset-normalizer ftfy
# optional
pip install confusable-homoglyphs

Python stdlib unicodedata covers most detection needs.

Edge cases

• Intentional script mixing — multilingual datasets (e.g. Japanese text with Latin product codes) are expected. Whitelist allowed script combinations rather than flagging everything.
• Domain-specific punctuation — code columns (e.g. identifiers) may legitimately use non-ASCII; don't auto-normalise without user confirmation.
• Round-trip-sensitive fields — cryptographic hashes, signatures, URLs with percent-encoding — never normalise; flag as "do not touch".
• ftfy limits — ftfy.fix_text is heuristic and occasionally over-corrects. Preview before applying at scale.
• Normalisation form downstream requirements — HF Datasets / JSON-LD / web APIs typically want NFC; some linguistic tools want NFD. Ask before picking.

Safety

Follow the backup policy in CONVENTIONS.md before any in-place mutation of text values.