How Tokens Are Counted: BPE and the Hidden Token Tax

Early NLP swung between two poles. Word-level tokenization is semantically clear, but English's hundreds of thousands of words plus morphology (look / looks / looking / looked) explode the vocabulary—and still can't handle unseen words (the OOV problem). Character-level tokenization can split anything, but sequences get too long, each character carries little information, and inference is painfully inefficient.

Modern LLMs took the middle road—subword tokenization: keep common words as whole Tokens, split rare ones into meaningful pieces. Vocabularies stay between 32k and 200k—no explosion, yet everything is representable.

A Token isn't simply a character or a word—it's a “subword” built from statistical patterns. By balancing vocabulary size and semantic expression, BPE became the foundation of modern LLM training and inference. (Figure: from the author's original share deck)

GPT-series models and Llama 3 all use BPE (Byte-Pair Encoding). The vocabulary isn't hand-crafted—it's “merged” out of the corpus:

At inference, look it up in reverse: a common word like "hug" is one Token; a rarer "bug" splits into ["b", "ug"]. The larger the vocabulary and the more “common” your text, the fewer Tokens—and the thinner the bill. That's why after GPT-4 moved to a 100k vocabulary, it specifically merged multi-level indent space sequences into single Tokens—code-generation efficiency jumped several times over.

Left: during BPE training, iteratively merge the highest-frequency byte pairs to build the vocabulary. Right: at inference, pre-tokenize by regex, then greedily merge by merge priority. (Figure: from the author's original share deck)

Tokenizers learn merge rules from their training mix, so efficiency depends on the exact model, language, script, punctuation, and content domain. Some multilingual or non-Latin inputs use more Tokens than an English equivalent, while other tokenizers narrow the gap—the practical “Token tax” is a measurement problem, not a universal multiplier.

These are teaching examples, not a universal language price table. For the same meaning, token counts can diverge enough to affect API spend and effective context length, so benchmark representative prompts with the tokenizer and model you will actually deploy.

Token boundaries depend on the model's vocabulary and training mix: compare representative inputs instead of assuming one language-wide multiplier. (Figure: from the author's original share deck)

Pick a sentence and compare an English version with a Chinese-language example using rough teaching estimates. For exact splits, use the tokenizer for your target provider—for example, the OpenAI Tokenizer.

English has a visible guardrail: before BPE merges, text is pre-tokenized on spaces, so merges stay inside words and Token boundaries often match linguistic intuition. Chinese, Japanese, Thai, and other scripts may not use spaces between words, so a tokenizer relies more heavily on statistical co-occurrence and its training coverage. The result can be either compact or surprisingly fragmented—measure the target workload.

One often-cited Chinese example is an unusual string discussed as a “glitch Token”: it became a single token in one model vocabulary because of corpus frequency. The lesson is broader than that one string: tokenizer behavior is statistical, model-specific, and sometimes unintuitive.

Space-delimited languages and scripts without spaces expose different tokenizer trade-offs; both should be checked on real product prompts. (Figure: from the author's original share deck)

Source: Adapted from Part 1 “What Makes Up a Token” of the author's internal team share “AI Token Cost Engineering Strategies.” For BPE details see OpenAI Tokenizer and the open-source tiktoken library. To review the underlying principles of tokenization and vocabulary, revisit LLM Fundamentals · Vocabulary & Training.