TWIL #004 - The Asian Language Tax in LLMs
Why LLM is more expensive for Asians
- #ai
- #llm
- #tokenization
A paper published on Zenodo quantified something that felt anecdotally true but hadn't been formally measured: Asian languages are structurally penalised by LLM tokenizers.
The problem is in how tokenizers split text into subword tokens. Latin-script languages map efficiently - roughly one token per word or morpheme. Non-Latin scripts like Burmese, Thai, or Chinese get fragmented far more aggressively. The result: the same semantic content costs up to 11.7x more tokens in Burmese on the cl100k_base tokenizer (used by GPT models). The median across 11 non-Latin scripts is 8.9x.
This isn't just a billing problem. It's a capability problem.
Recall performance - how well a model retrieves information from its own context window - collapses to 0-2 out of 10 by 4,000 tokens for 4 of the 5 tested models when using non-Latin scripts. Effectively, Asian language users get a much smaller usable context window for the same nominal limit.
One bright spot: Gemini 2.5 achieved 84.5% recall, which suggests this is a solvable engineering choice, not an inherent limitation of the technology.
For anyone building AI products for non-English markets, tokenizer choice is not a footnote - it's a first-class design decision.