thai-text-processing

Thai Text Processing (ประมวลผลข้อความภาษาไทยในโค้ด)

Overview

Thai is written without spaces between words, uses combining tone marks, and does not sort by codepoint. Naive code that works for English silently corrupts Thai: "ฉันกินข้าว".split(" ") returns one token, ORDER BY name puts vowels in the wrong place, and LEFT(name, 20) may chop a syllable mid-character and render ◌. This skill is the checklist of things to fix.

When to use

• ตัดคำภาษาไทย (segmentation) สำหรับ search, autocomplete, n-grams
• Unicode normalization (NFC) ก่อนเก็บ / เทียบสตริง
• จัดเรียง (sort / collate) ชื่อภาษาไทยใน DB หรือ list
• ทำ romanization สำหรับ slug, passport-style spelling, academic citation
• ตั้งค่า full-text search index ใน ES / Postgres / MySQL
• ตัดข้อความ (truncate) แบบไม่ทำให้ glyph แตก
• เลือกไลบรารี: PyThaiNLP vs ICU vs nlpO3 vs Lucene Thai analyzer

1. The core problem: no spaces

"ฉันกินข้าว".split(" ")  # ["ฉันกินข้าว"]  ← one mega-token

This breaks:

• full-text search (ILIKE '%กิน%' mostly works, ILIKE '%ข้าว%' mostly works, but ranking, stemming, autocomplete all need proper tokens)
• word counting / "how long is this in words"
• text wrap / hyphenation
• n-gram features for ML
• highlight-on-search snippets

You must run a segmenter before any token-based operation.

2. Word segmentation toolkit

Tool	Language	Notes
PyThaiNLP word_tokenize	Python	De-facto default. Engines: newmm (default, dict-based), longest, attacut (CNN), deepcut (LSTM, slow but accurate)
nlpO3	Python via Rust	Fast newmm-style; permissive license
ICU BreakIterator	C / Java / Python (pyicu)	Built into many platforms; decent for word-break
thai-segmenter	TypeScript	Browser-friendly; dictionary-based
Lucene Thai analyzer	Java	Use for Elasticsearch / OpenSearch indexing
SentencePiece / BPE	Any	For ML training; not for human-facing tokens

Picking rules of thumb:

• Server Python: PyThaiNLP newmm (good enough, fast, MIT)
• Need higher recall on OOV / colloquial: attacut or deepcut (slower)
• Frontend JS: thai-segmenter or call ICU via WebAssembly
• Search index: dedicated Thai analyzer in ES (thai tokenizer) — do not segment client-side and ship tokens to ES
• Production at scale: nlpO3 for raw throughput

See examples/segmentation.py for runnable code.

3. Unicode normalization (NFC)

Thai displays one "visual character" out of up to 3 codepoints: base consonant + above-vowel + tone mark. The same word can be encoded multiple ways:

import unicodedata
a = "แสง"                              # may be precomposed
b = unicodedata.normalize("NFD", a)    # explicitly decomposed
a == b                                  # False
unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b)  # True

Rules:

• Always normalize to NFC before storing or comparing.
• Database column: pre-normalize in application, OR use a generated column / trigger.
• "Length in user-visible characters" = grapheme clusters, not codepoints. Use regex library \X (Python) or Intl.Segmenter("th", {granularity: "grapheme"}) (JS).
• len("ก่") in Python returns 2 (codepoints) — but the user sees 1 character.

See examples/normalize.py.

4. Romanization standards

System	Use case	Lossless?	Example: สวัสดี
RTGS (Royal Thai General System)	Signs, passports, SEO slugs	No (drops tone, vowel length)	sawatdi
ISO 11940	Academic, archives	Yes (one-to-one, ugly)	s̄wạs̄dī
ISO 11940-2	Phonetic, prettier	No	sawatdi
IPA	Phonetic transcription	Approximate	/sā.wàt.dīː/

Use RTGS for slugs and English-readable transliteration of names. PyThaiNLP exposes pythainlp.transliterate.romanize(text, engine="royin") for RTGS-style output. For deterministic batch slugs, also strip non-[a-z0-9-] after romanization.

5. Sorting / collation

Thai does NOT sort by codepoint. The traditional rule: sort by base consonant first, then vowel, then tone mark. Codepoint sort puts leading vowels (เ, แ, โ, ใ, ไ) AFTER consonants, which is wrong.

Place	How
PostgreSQL (ICU build)	name TEXT COLLATE "th-TH-x-icu"
MySQL 8	COLLATE utf8mb4_thai_520_w2 (Unicode 5.20, weights v2)
Python (no DB)	pyicu Collator with Locale("th_TH"), or PyThaiNLP's thai_strftime/collate
JavaScript	Intl.Collator("th", {sensitivity: "base"})
Elasticsearch	icu_collation_keyword field with language: "th"

-- PostgreSQL example
CREATE COLLATION thai_icu (provider = icu, locale = 'th-TH');
SELECT name FROM users ORDER BY name COLLATE "th-TH-x-icu";

Default UTF-8 / binary sort is wrong — verify by sorting ["แอน","กิน","โต","ไกล"] and confirming Thai dictionary order, not codepoint order.

6. Truncation safety

Truncating mid-character (codepoint vs grapheme) can leave dangling combining marks rendered as ◌. Rules:

• Truncate at grapheme cluster boundary. Python: regex library \X. JS: Intl.Segmenter with granularity: "grapheme".
• For UI ellipsis (e.g. "show first 30 chars + …"), first segment by word using PyThaiNLP, then count graphemes, then append … — don't cut mid-syllable.
• Database VARCHAR(50) counts bytes in some collations and characters in others. Test before assuming "50 chars" matches user perception.

See examples/normalize.py for grapheme-safe truncation.

7. Database / Elasticsearch indexing

Elasticsearch / OpenSearch:

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "thai",
        "search_analyzer": "thai"
      }
    }
  }
}

PostgreSQL full-text: the bundled tsvector configs don't ship a Thai dictionary. Options:

• Segment in application using PyThaiNLP, store space-separated tokens in a tsvector column with simple config
• Use pg_search / paradedb with a Thai tokenizer
• Use a sidecar Elasticsearch

Don't rely on ILIKE '%term%'. Without segmentation it does substring match — works for very short queries, fails for multi-word, misses synonyms, can't rank.

8. LLM tokenizers and Thai text

Modern LLMs (Claude, GPT, Gemini, Llama) use BPE / SentencePiece tokenizers trained primarily on English. Thai gets a rough ride:

• Characters are frequently split mid-cluster — base consonant in one token, tone mark or above-vowel in another. The model still "reads" the word, but boundaries are arbitrary.
• Thai consumes ~2–4× more tokens than English for equivalent semantic content. A 1,000-token English prompt becomes ~2,500–4,000 tokens in Thai.
• Truncating by byte / character length can mid-split a glyph — โ (leading vowel) ends up in one chunk and its consonant in the next, both render as ? or ◌ in downstream UIs.
• Embeddings on raw Thai degrade vs. word-segmented input — semantic search recall drops noticeably when the tokenizer fragments words at non-linguistic boundaries.
• Char-counting tooling on top of LLM output often miscounts grapheme clusters (see §6).

Practical advice:

• Pre-segment with PyThaiNLP (word_tokenize or Tokenizer(engine="newmm")) before semantic search, RAG chunking, or embedding generation.
• Budget tokens at ~3× English baseline when sizing context windows and API cost.
• Truncate at word boundaries (segment first, then cut), never by byte length on raw Thai.
• For chunking long Thai documents for RAG, chunk on segmented word boundaries — naive chunk_size=512 character splits will fragment compound nouns and tank retrieval.

9. TIS-620 encoding (legacy)

TIS-620 is the pre-Unicode Thai character encoding still found in:

• Old government and state-enterprise databases
• Banking core systems
• Some POS / inventory / accounting software
• Legacy CSV / Excel exports from Thai vendors
• Old PDF text layers (especially scanned-then-OCR'd)

It is a single-byte encoding: bytes 0x00–0x7F are ASCII, bytes 0xA1–0xFB map to Thai characters. Microsoft's variant is Windows-874 / cp874 — TIS-620 plus a few extra symbols (curly quotes, euro sign, etc.).

Pitfalls:

• Opening a TIS-620 file as UTF-8 yields mojibake like àÅÂä·Â.
• Saving the mojibake back as UTF-8 double-encodes — recovery requires decoding as UTF-8, re-encoding as Latin-1, then decoding as TIS-620.
• Some old systems use 0xA0 (non-breaking space) as a Thai consonant by mistake — strict TIS-620 decoders raise on it.
• File-encoding auto-detection (chardet, charset-normalizer) is unreliable on short Thai strings — confirm by inspecting bytes, not by guessing.

Recipe (Python):

# Read a legacy file as TIS-620 or cp874 and write back as UTF-8
with open("legacy.csv", "rb") as f:
    raw = f.read()

# Try cp874 first (superset of TIS-620, handles MS exports too)
try:
    text = raw.decode("cp874")
except UnicodeDecodeError:
    # Fall back, replacing the stray 0xA0 etc.
    text = raw.decode("cp874", errors="replace")

with open("clean.csv", "w", encoding="utf-8") as f:
    f.write(text)

# Recover double-encoded mojibake (UTF-8 wrapper around TIS-620 bytes)
def fix_double_encoded(s: str) -> str:
    return s.encode("latin-1").decode("cp874")

For Pandas: pd.read_csv("legacy.csv", encoding="cp874"). For requests / HTTP: check Content-Type: text/html; charset=tis-620 — old Thai gov sites still serve it.

Common mistakes

• .split(" ") to count Thai words → always returns 1 — use a segmenter
• ORDER BY name without collation → vowels and tone marks in codepoint order, not dictionary order
• Storing un-normalized text → WHERE name = 'แสง' misses rows that were saved decomposed
• LEFT(text, 20) for "first 20 chars" → counts bytes or codepoints, not graphemes; renders ◌
• "Username max 20 chars" enforced by codepoint count → users can submit 60-byte usernames with combining marks
• Using RTGS for accurate phonetic transcription → it drops tone and vowel length; use IPA
• Shipping client-segmented tokens to Elasticsearch → bypasses the analyzer; mismatched query-time tokenization
• Treating "ๆ" (mai-yamok, repetition mark) as a separate word → it's an in-word marker; PyThaiNLP handles it correctly, .split() doesn't
• Forgetting that ​ (zero-width space, U+200B) is sometimes injected by CMSs for word break hints — strip on input or normalize
• Sizing LLM API budgets by English character count → Thai costs ~3× more tokens; the prompt fits in the context window until it suddenly doesn't
• Chunking Thai documents for RAG by raw character count → fragments words mid-cluster, embeddings lose semantic meaning, retrieval recall drops
• Trusting chardet to identify TIS-620 vs UTF-8 on short strings → it guesses wrong on names and headlines; check the byte range manually or try cp874 decode first
• Reading a TIS-620 file with the default UTF-8 codec and "fixing" the mojibake by hand-editing → you've now double-encoded; always re-decode from raw bytes