Tokenization: why counting the letters in strawberry can trip up a model

Ask a language model “How many r’s are in strawberry?” and for a long time a lot of them would confidently tell you two. There are three. It became a running joke, because the same model could write you a decent essay about strawberries.

The thing is, the model never saw the word the way you just did. It didn’t see ten letters. It saw a short list of numbers. Once you see what those numbers are, the strawberry failure stops being mysterious, and a few other weird things about language models start making sense too, like why they’re bad at reversing strings, why APIs charge “per token”, and why the same sentence costs more in Hindi than in English.

What the model actually sees

Before any text reaches the neural network, a separate program called a tokenizer chops it into pieces and replaces each piece with an integer ID. The model only ever works with those IDs. It has a big table that maps each ID to a vector, and everything after that is math on vectors.

You can look at this directly. OpenAI publishes its tokenizers in an open source library called tiktoken.¹ Here’s the question itself, run through cl100k_base, the tokenizer behind GPT-3.5 and GPT-4:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("How many r's are in strawberry?")
print(len(ids))                                    # 8
print([enc.decode_single_token_bytes(t) for t in ids])
# [b'How', b' many', b' r', b"'s", b' are', b' in', b' strawberry', b'?']

Eight tokens. And look at the seventh one. strawberry, with the leading space, is a single token. Its ID is 73700. That’s what the model gets. Not s t r a w b e r r y. Just 73700.

It gets weirder. If the word shows up without a space in front, like at the very start of a message, it gets split differently:

So depending on context the model sees one token, or str + aw + berry, or, with the newer o200k_base tokenizer used by GPT-4o and later models,¹ st + raw + berry. None of those pieces carry the label “contains one r” or “contains two r’s”. The spelling is hidden inside the ID.

Where tokens come from: byte pair encoding

Why str and aw? Why is strawberry its own token but blueberry gets split into blue + berry in cl100k_base? The answer is an algorithm called byte pair encoding, or BPE.

BPE started as a data compression trick. Philip Gage described it in 1994: find the most common pair of adjacent bytes, replace it with a new symbol, repeat.² In 2016, Rico Sennrich, Barry Haddow, and Alexandra Birch adapted it for neural machine translation, so a model with a fixed vocabulary could still handle rare words by breaking them into smaller pieces that it had seen before.³ That idea underlies many widely used tokenizers, though other methods exist.

Training a BPE tokenizer goes like this:

1. Start with every word split into single characters.
2. Count every adjacent pair across the whole training corpus.
3. Merge the most common pair into one new symbol and add it to the vocabulary.
4. Repeat until the vocabulary is as big as you want.

Here’s a toy version you can run. The word counts are the little example corpus from the Sennrich paper.³

from collections import Counter

# word -> how often it appears, each word split into characters
corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
words = {tuple(w): n for w, n in corpus.items()}

for step in range(1, 6):
    pairs = Counter()
    for symbols, n in words.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[a, b] += n
    (a, b), count = pairs.most_common(1)[0]
    print(f"merge {step}: {a!r} + {b!r} (seen {count} times)")
    merged = {}
    for symbols, n in words.items():
        out, i = [], 0
        while i < len(symbols):
            if symbols[i:i + 2] == (a, b):
                out.append(a + b)
                i += 2
            else:
                out.append(symbols[i])
                i += 1
        merged[tuple(out)] = n
    words = merged

print(list(words))

When I ran it, the output was:

merge 1: 'e' + 's' (seen 9 times)
merge 2: 'es' + 't' (seen 9 times)
merge 3: 'l' + 'o' (seen 7 times)
merge 4: 'lo' + 'w' (seen 7 times)
merge 5: 'n' + 'e' (seen 6 times)
[('low',), ('low', 'e', 'r'), ('ne', 'w', 'est'), ('w', 'i', 'd', 'est')]

After five merges, low is a single token and est is a reusable suffix. Nobody told the algorithm that “-est” means something in English. It just showed up a lot. Scale this to a huge chunk of the internet and tens of thousands of merges, and you get pieces like berry and whole words like strawberry. Common strings become one token. Rare strings get built from smaller parts. That’s the point: frequent stuff is cheap, and nothing is ever impossible to represent.

Bytes, spaces, and categories

Two details from GPT-2 explain a lot of what you see in modern tokenizers.

First, GPT-2 ran BPE on raw UTF-8 bytes instead of Unicode characters. The paper points out that if you start from Unicode characters you need a base vocabulary of over 130,000 symbols before you merge anything, while bytes only need 256.⁴ With bytes, any string in any language can be encoded. That will matter a lot later in this post.

Second, they stopped BPE from merging across character categories, so you don’t end up with separate tokens for dog., dog!, and dog?, with one exception: a space is allowed to stick to the front of a word.⁴ That’s why strawberry with a space and strawberry without one are different tokens. The tokenizer first splits text with a regular expression into rough chunks (words with their leading space, runs of digits, runs of punctuation), and BPE only merges inside those chunks. You can read the exact patterns in the tiktoken source.¹

tiktoken and SentencePiece

You’ll mostly run into two families of tools. tiktoken is OpenAI’s fast BPE implementation, and it ships the actual vocabularies: about 50,000 tokens for GPT-2, about 100,000 for cl100k_base, about 200,000 for o200k_base (the names give it away).¹ SentencePiece, from Taku Kudo and John Richardson at Google, trains subword models “directly from raw sentences” without needing a language-specific pre-tokenizer, which makes it popular for multilingual models.⁵ It supports BPE and another method called unigram. The details differ, but the big picture is the same: text goes in, a list of integer IDs comes out, and the network receives token IDs rather than a separate character-by-character spelling.

Why can counting the r’s be difficult?

Put yourself in the model’s position. You get the ID 73700. To answer “how many r’s”, you’d need to know that 73700 is spelled s-t-r-a-w-b-e-r-r-y. Nothing in the input tells you that. The only way to know is if, somewhere in training, you picked up the spelling of that token indirectly, from text where people spell words out, from typos, from rhymes, stuff like that.

Models do learn a surprising amount of this. That’s why they can often spell a word if you ask. But “know the spelling” and “count a letter inside a token you can’t see into” are different jobs.

Here’s the part that complicates the simple story. In late 2024, Tairan Fu and colleagues tested this directly. They found that models can recognize which letters are in a word but struggle to count them. Errors were not explained by how frequent the word or its tokens were in training data. Errors tracked word length and, most strongly, how many letters repeat. Most models couldn’t count a letter that appears more than twice.⁶ In strawberry the letter r appears three times, two of them back to back. That’s exactly the hard case.

So my read is: tokenization makes the letters invisible, and counting makes it worse. The model has to reconstruct the spelling from memory and then do a little counting task in its head, all inside one forward pass per output token. Two hard things stacked.

The workaround also makes sense once you see it. Ask a recent model and it may well get this right, often by spelling the word out first, one letter per line. That’s not the model getting smarter at reading tokens. Writing s, t, r, … turns each letter into its own token, so the spelling is explicit in the context. That can help counting, though it does not guarantee a correct answer.

The same problem, other places

Once you know about tokens, you start seeing them everywhere. Andrej Karpathy has a long lecture where he builds the GPT tokenizer from scratch, and he traces a bunch of classic model weirdness back to it, including trouble reversing strings and doing arithmetic.⁷

Numbers are a good example. cl100k_base splits runs of digits into chunks of up to three, left to right.¹ So 12345678 becomes 123 + 456 + 78. GPT-2’s tokenizer had no such rule and split the same number into 123 + 45 + 678. In both cases the chunks don’t line up with place value. To add 1234 + 5678 the model sees 123, 4, +, 567, 8. You can still learn arithmetic like that, but you’re doing it the hard way.

Tokens are also the bill

Tokens aren’t only how the model reads. They’re how everything gets measured. Context windows are measured in tokens. Generation speed is tokens per second. And API pricing is per token, for input and output. OpenAI’s rule of thumb for English is about 4 characters or three quarters of a word per token, and their help page warns that the ratio can be worse for other languages, which makes those languages more expensive to use.⁸

That means anything that makes text tokenize badly is a tax. You pay more, you fit less into the context window, and the model has to do more steps to read the same content.

Code used to be expensive

Code is a nice example of a tax that got fixed. Here’s a tiny Python function:

def count_r(word):
    total = 0
    for ch in word:
        if ch == "r":
            total += 1
    return total

That’s 115 characters. With GPT-2’s tokenizer it’s 59 tokens. With cl100k_base it’s 34. Most of the difference is indentation. When I printed the GPT-2 tokens, every single space of indentation was its own token, so an 8-space indent cost 8 tokens before you’d written any code. The newer tokenizers have tokens for runs of whitespace, so the same indent is one token. For comparison, a plain English sentence explaining the same function (132 characters) is 34 tokens in all three tokenizers.

So code doesn’t have to be expensive. It was expensive because the tokenizer was trained mostly on prose and never learned that four spaces in a row is a very common string.

Why Hindi costs more

Last October I gave a talk at a school in Hyderabad where some students were in English-medium classes and some were in Hindi-medium classes, so I switched back and forth between the two languages the whole time (I wrote about it here). If I had wanted to give those same students an AI tutor, the Hindi-medium ones would have been paying a hidden tax on every message.

That connection is honestly what made me want to measure it. Here’s a sentence about the talk, in English and in Hindi, run through three OpenAI tokenizers:

import tiktoken

english = "After the talk, the students asked very good questions."
hindi = "भाषण के बाद छात्रों ने बहुत अच्छे सवाल पूछे।"

for name in ["gpt2", "cl100k_base", "o200k_base"]:
    enc = tiktoken.get_encoding(name)
    print(f"{name:12} English: {len(enc.encode(english)):3}  Hindi: {len(enc.encode(hindi)):3}")

gpt2         English:  11  Hindi:  68
cl100k_base  English:  11  Hindi:  47
o200k_base   English:  11  Hindi:  13

With the GPT-2 tokenizer, the Hindi sentence takes six times as many tokens as the English one. With cl100k_base, about four times. With o200k_base, they’re basically even.

The reason goes back to bytes. Devanagari characters take 3 bytes each in UTF-8. The Hindi sentence is 44 characters but 116 bytes, while the English one is 55 characters and 55 bytes. A byte-level tokenizer trained mostly on English never saw enough Hindi to learn merges for it, so it falls back to tiny pieces, sometimes single bytes that aren’t even a whole character. When I printed the cl100k_base tokens for the Hindi sentence, a lot of them were fragments like \xe0\xa4 that only make sense glued to the next token.

What the research says

My one sentence is just an illustration. Aleksandar Petrov and coauthors did this properly at NeurIPS 2023, using FLORES-200, a set of the same 2,000 sentences translated into 200 languages.⁹ They define a language’s tokenization premium as how many tokens it needs relative to English for the same content. For Hindi, the premium was 7.46 with the GPT-2 tokenizer and 4.79 with cl100k_base.⁹ My single sentence (6x and about 4x) lands close to that. Across all languages, they found differences of up to 15 times. Shan, a language spoken in Myanmar, needed about 15 times as many tokens as English with the ChatGPT and GPT-4 tokenizer.⁹ They also found the gap persists even in tokenizers built specifically to be multilingual.⁹

Around the same time, Orevaoghene Ahia and coauthors looked at what this means for money, using OpenAI’s API across 22 languages. Speakers of many languages pay more for the same content and also get worse results, and many of those speakers live in places where the API is already less affordable.¹⁰ That extra cost can be especially burdensome where the API is already less affordable. It’s just a side effect of what text the tokenizer was trained on.

The good news is it’s fixable, at least partly. When OpenAI launched GPT-4o in May 2024, they highlighted the new tokenizer’s compression across languages, and for their Hindi sample text it used 2.9 times fewer tokens than before.¹¹ My sentence above shows the same jump. Bigger vocabulary, more non-English training text for the tokenizer, and the token-count gap narrows substantially in this example.

What I take from this

I find it surprising how much can depend on the way text is split up before the model even starts working. It affects the task we are asking it to do and the cost of sending that text. The language differences make me especially curious about how much better this could become for people using languages other than English.

For counting letters, I’d like to compare words with different token splits and different amounts of repetition. Fu et al. found that repeated letters mattered more than token frequency in their experiments.⁶ Testing those factors together would help separate what the tokenizer contributes from the difficulty of counting itself.

Karpathy’s lecture offers a way to build a tokenizer and inspect these choices directly.⁷ Even looking at a few familiar sentences can raise useful questions about what we are giving the model and what we expect it to recover.

Sources