How LLM Tokenizers Work: BPE, WordPiece and Why Tokens Aren't Words
Why AI models read numbered word pieces instead of letters, and how that choice shapes cost, limits and quirks

Ask a chatbot to count the letter "r" in "strawberry" and it may stumble. Paste the same paragraph into two AI models and you can get two different bills. Both puzzles have the same cause: language models never see your letters. They see tokens.
Tokens are the unit everything in an AI model is measured in: context windows, rate limits, pricing and output length. This guide explains how LLM tokenizers work, from the 1994 compression trick behind byte pair encoding (BPE) to the WordPiece, Unigram and SentencePiece variants, and what it all means for the prompts and budgets you manage.
What is a token?
A model's neural network only does math on numbers, so text has to be converted into a sequence of integers before the model can read it. A tokenizer does that conversion. OpenAI's help center describes a token as something that can be a single character, part of a word, a whole word or a punctuation mark, and notes that spaces affect the split too.
Designers had three broad choices for the unit of text:
- Whole words. Easy to understand, but the vocabulary explodes, and any word not seen in training (a new product name, a typo) becomes an unknown token.
- Single characters. Nothing is ever unknown, but sequences get very long, and the model must learn spelling from scratch.
- Subwords. The middle ground nearly every modern model uses: common words stay whole, rarer words are built from reusable pieces.
OpenAI's rule of thumb for English is that one token is roughly four characters, or about three-quarters of a word, so 100 tokens is around 75 words. Treat that as a rough estimate only; code, numbers and other languages can behave very differently.
How byte pair encoding (BPE) works
BPE did not start in AI. Philip Gage described it in 1994 as a data compression method: find the most common pair of adjacent bytes, replace it with a new symbol, and repeat. It's a close cousin of the repetition-hunting ideas in our guide to how data compression works.
In 2016, Rico Sennrich, Barry Haddow and Alexandra Birch at the University of Edinburgh adapted the idea for neural machine translation, so a model could handle rare and unseen words as sequences of known subword units. Training a BPE tokenizer works like this:
- Split a large training corpus into words and count how often each appears.
- Start with a vocabulary of individual characters (or bytes).
- Count every adjacent pair of symbols, weighted by word frequency.
- Merge the most frequent pair into a new symbol and add it to the vocabulary.
- Repeat until the vocabulary reaches the target size, a setting chosen in advance.
A tiny worked example
Hugging Face's tokenizer documentation uses a toy corpus of five words with counts. Here is a short Python version of the training loop and what it learns in three merges:
from collections import Counter
corpus = {"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5}
words = {tuple(w): n for w, n in corpus.items()}
def merge_once(words):
pairs = Counter()
for syms, n in words.items():
for a, b in zip(syms, syms[1:]):
pairs[(a, b)] += n
best = pairs.most_common(1)[0][0]
out = {}
for syms, n in words.items():
new, i = [], 0
while i < len(syms):
if syms[i:i+2] == best:
new.append(syms[i] + syms[i+1]); i += 2
else:
new.append(syms[i]); i += 1
out[tuple(new)] = n
return best, out
for _ in range(3):
best, words = merge_once(words)
print(best, list(words))
# ('u', 'g') -> h+ug, p+ug, p+u+n, b+u+n, h+ug+s (seen 20 times)
# ('u', 'n') -> ... p+un, b+un ... (seen 16 times)
# ('h', 'ug') -> hug, p+ug, p+un, b+un, hug+s (seen 15 times)
To tokenize new text, the tokenizer replays those merges in the order it learned them. That's why BPE is deterministic: the same input always produces the same tokens.
Byte-level BPE: never an unknown character
Character-based BPE still fails on characters it never saw in training. GPT-2 solved this by starting from raw bytes instead of characters. Since any text can be expressed in UTF-8 bytes, there are only 256 possible starting symbols, and nothing is ever out of vocabulary. According to Hugging Face, GPT-2's vocabulary of 50,257 is exactly 256 byte tokens, 50,000 learned merges and one special end-of-text token.
OpenAI's open-source tiktoken library, which implements the tokenizers for its models, lists the properties that make byte-level BPE attractive: it is reversible and lossless, it works on any text, it compresses (each token averages about 4 bytes in practice), and it lets the model see common subwords again and again. Its README gives the example of "encoding" splitting into "encod" and "ing".
WordPiece, Unigram and SentencePiece
BPE is the most common approach, but not the only one. Hugging Face's Transformers library supports three subword algorithms, and SentencePiece is a toolkit that wraps two of them.
| Approach | How it picks pieces | Known users |
|---|---|---|
| BPE | Merges the most frequent adjacent pair, bottom-up | GPT family (byte-level) |
| WordPiece | Merges the pair that appears together far more often than chance predicts | BERT, DistilBERT, Electra |
| Unigram | Starts large and prunes pieces that matter least; picks the most probable split | T5, BigBird, Pegasus |
| SentencePiece | Runs BPE or Unigram on raw text, treating the space as a symbol (▁) | Many multilingual models |
WordPiece scores a candidate pair by dividing how often the pair occurs by the product of how often each half occurs alone. It marks pieces that continue a word with ##, so BERT turns "gpu" into gp + ##u. Unigram is probabilistic: the same word could split several ways, and it can even sample different splits during training. SentencePiece exists because languages such as Chinese and Japanese don't put spaces between words, so it treats the input as a raw stream and stores the space itself as "▁".
Why tokenization matters in practice
Tokenization quietly shapes cost, limits and quality:
- Pricing. APIs bill per token, often at different rates for input, output and cached input. Reasoning models may also burn hidden reasoning tokens that are billed as output.
- Context limits. A model's context window and output cap are counted in tokens, not words or pages.
- Spaces and capitals. OpenAI points out that "red", "Red" and " red" (with a leading space) can all be different tokens.
- Non-English text. OpenAI notes that other languages can have different relationships between characters, words and tokens, which can raise costs.
- Spelling and arithmetic. When "strawberry" arrives as a couple of chunks rather than ten letters, letter-level questions get harder for the model.
- Semantic search. Tokens are what get turned into vector embeddings, so chunk sizes for retrieval are usually set in tokens too.
Counting tokens yourself
For OpenAI models you can count locally with tiktoken. Other providers publish their own counting tools or report usage in API responses.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
ids = enc.encode("Tokenizers turn text into numbers.")
print(len(ids)) # token count
print([enc.decode([i]) for i in ids]) # the pieces
assert enc.decode(ids) == "Tokenizers turn text into numbers."
pip install tiktoken. Use the encoding that matches your target model.Key takeaways
- Language models read token IDs, not letters; a tokenizer converts text to those IDs and back.
- Most modern models use subword tokens: common words stay whole, rare words are assembled from pieces.
- BPE learns a vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE starts from 256 bytes so nothing is ever unknown.
- WordPiece, Unigram and SentencePiece are alternative recipes used by BERT, T5 and many multilingual models.
- Token counts drive price, context limits and quality, and they differ between models, so measure rather than guess.
Frequently asked questions
How many words is 1,000 tokens?
For typical English, roughly 750 words, using OpenAI's estimate that a token is about three-quarters of a word. Code, numbers and other languages can differ a lot, so count with a tokenizer for anything that matters.
Do all AI models use the same tokenizer?
No. Each model family trains its own vocabulary, and even models from the same company can use different encodings. The same text can produce different token counts on different models.
Why do LLMs struggle to count letters in a word?
The model receives a word as one or a few subword tokens rather than individual letters, so it has no direct view of the spelling and must infer it from patterns learned in training.
What is the difference between BPE and WordPiece?
Both build a vocabulary by merging pairs. BPE merges whichever pair is most frequent; WordPiece favours pairs that occur together much more often than their separate frequencies would predict.












Comments
No comments yet — be the first to share your thoughts.
Join the discussion