Tokenisation
Lesson 1 of 8
1Tokenisation
The big pictureA model reads neither letters nor words, only numbers. The tokenizer decides how text is cut into pieces and each piece into a number, and that decision sets what you pay, how much fits in the context, and why an Arabic sentence costs more than its English translation.
What the picture shows: Watch the Arabic sentence break into 12 pieces, mostly single letters, while every English word in the example became one token. That gap is what you pay for in price and in context space.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A model reads neither letters nor whole words but pieces called tokens: a common word is one token, a rare word is cut into smaller pieces. Because most of what models learned from is English, an Arabic sentence is usually cut into more pieces, so it costs more and takes more room.
BPE starts from characters and merges the most frequent pairs until the vocabulary size is reached, and SentencePiece works on raw text without pre-splitting on spaces. Token count sets cost and context length, and Arabic, with its rich morphology, yields more tokens per word in an English-trained vocabulary.
Before You Start
🔢Arrays with NumPy
You know a model works on numbers in arrays, not on text, so everything that goes in must become numbers first.
🐍Counting in Python
You can use Counter and zip to count adjacent pairs in a list, which is all you need to write BPE by hand.
The Core Idea
In this lesson you see why text is cut into pieces, build BPE by hand, count bytes, and finally understand why Arabic costs more tokens.
1Why we cut text into pieces
A model needs a fixed vocabulary of numbers. Cutting letter by letter gives a vocabulary in the tens but very long sequences, and cutting word by word gives short sequences but millions of words and unknown ones.
The answer is in between: pieces bigger than a letter and smaller than a word. Common things become one token, rare ones several pieces, and nothing is unknown.
Look at the picture: the tokenizer is the middle step, built once before training and then frozen with the model. So how it cuts your sentence decides what you pay.
2BPE: merge the most frequent pair
The best-known way to build these pieces is called BPE. It does not take the pieces from a dictionary; it discovers them in the text itself: what often appears together becomes one piece.
Start with every word as separate letters plus an end marker </w>. Count every adjacent pair, merge the most frequent into one symbol, and repeat.
from collections import Counter
words = ["lower", "lowest", "newer", "newest", "wider", "widest"]
seqs = [list(w) + ["</w>"] for w in words]
def merge_once(seqs):
pairs = Counter()
for s in seqs:
pairs.update(zip(s, s[1:]))
(a, b), n = pairs.most_common(1)[0]
out = []
for s in seqs:
t, i = [], 0
while i < len(s):
if i + 1 < len(s) and s[i] == a and s[i + 1] == b:
t.append(a + b)
i += 2
else:
t.append(s[i])
i += 1
out.append(t)
return (a, b, n), out
for step in range(6):
(a, b, n), seqs = merge_once(seqs)
print(f"merge {step + 1}: {a!r} + {b!r} (seen {n} times)")
print(seqs[1])
print(seqs[4])
After only six merges lowest is two pieces, lowe and st</w>, because word-final st appeared three times. wider is still letters, since its pairs did not repeat enough. A real tokenizer does exactly this tens of thousands of times over billions of words, so common words become one token and rare ones several pieces.
Look at the picture: the first merge was w with e, because it appeared 4 times. Notice that a merge can build on an earlier one, like st and then st</w>.
3Bytes: an Arabic letter is two bytes
Before any merge, where does a tokenizer start? Look at the picture: three sentences with about the same meaning, and the Arabic starts from a longer sequence, which diacritics stretch a lot.
Most modern tokenizers work on bytes, not characters, so no symbol is ever unknown. And an Arabic letter in UTF-8 is two bytes, an English one is one.
samples = {
"English": "The model reads the text",
"Arabic": "النموذج يقرأ النص",
"Arabic with diacritics": "النَّموذَجُ يَقرَأُ النَّصَّ",
}
for name, text in samples.items():
print(f"{name:24} chars={len(text):3} utf8 bytes={len(text.encode('utf-8')):3}")
An Arabic sentence shorter in characters is longer in bytes, and diacritics nearly double it. If the tokenizer did not see enough Arabic while it was built, it will not merge these bytes into large pieces, and you pay for almost every byte.
4Why Arabic costs more
A tokenizer merges what it sees most. If it was built on mostly English text, it merged large English pieces and left Arabic in small ones.
The number that measures this is fertility: the average tokens per word. A good English tokenizer gives about 1.3, and older tokenizers can give Arabic 3 or 4.
We train a small BPE on text that is nine tenths English and one tenth Arabic, then tokenise a sentence from each language with roughly the same meaning.
from collections import Counter
def train_bpe(text, merges):
seqs = [list(w) + ["_"] for w in text.split()]
rules = []
for _ in range(merges):
pairs = Counter()
for s in seqs:
pairs.update(zip(s, s[1:]))
if not pairs:
break
(a, b), _ = pairs.most_common(1)[0]
rules.append((a, b))
seqs = [apply(s, a, b) for s in seqs]
return rules
def apply(s, a, b):
t, i = [], 0
while i < len(s):
if i + 1 < len(s) and s[i] == a and s[i + 1] == b:
t.append(a + b); i += 2
else:
t.append(s[i]); i += 1
return t
def tokenize(text, rules):
total = 0
for w in text.split():
s = list(w) + ["_"]
for a, b in rules:
s = apply(s, a, b)
total += len(s)
return total
english = "the student reads the lesson and the teacher reads the answer " * 20
arabic = "الطالب يقرأ الدرس والمعلم يقرأ الجواب " * 20
rules = train_bpe(english * 9 + arabic, merges=40) # trained mostly on English
test_en = "the teacher reads the lesson"
test_ar = "المعلم يقرأ الدرس"
print("English tokens:", tokenize(test_en, rules), "for", len(test_en.split()), "words")
print("Arabic tokens: ", tokenize(test_ar, rules), "for", len(test_ar.split()), "words")
Five English words became five tokens, one per word. Three Arabic words became 12 tokens, four per word. This is exactly what happens in large tokenizers, in different proportions: the language seen more gets its pieces merged more.
The result is more tokens for the same meaning: a higher price, a context that fills faster, and slower generation.
How It Works, Step by Step: how a BPE tokenizer is built
Start with every word as separate letters or bytes, plus an end-of-word marker such as </w>.
Count every adjacent pair across the whole text, such as w and e, seen together 4 times.
Merge the most frequent pair into one symbol, and add that merge to the list of rules.
Repeat until the vocabulary reaches its target size, then apply the same rules in order to any new text.
Going deeper
The useful number for comparison is fertility: the average number of tokens per word. A good English tokenizer gives about 1.3, and older tokenizers can give Arabic 3 or 4. Newer models that added a lot of Arabic to their vocabulary come much closer, which is why the same Arabic request costs differently from one provider to the next.
| Method | Vocabulary size | Sequence length | Unknown word |
|---|---|---|---|
| Characters | Tens | Very long | None |
| Words | Hundreds of thousands | Short | Many |
| Byte-level BPE | 32k to 200k | Medium | None |
Worked example
You have three ways to split text: letter by letter (a small vocabulary but very long sequences), word by word (short sequences but millions of words and words the model has never seen), or pieces in between. Every modern model picks the third, and the best-known way to build it is called BPE.
Example: Counting tokens with a real tokenizer
In your project use the model's own tokenizer. This needs transformers installed and the tokenizer downloaded, so it is here for reading.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
for text in ["The model reads the text", "النموذج يقرأ النص"]:
ids = tok.encode(text)
print(len(text.split()), "words ->", len(ids), "tokens:", tok.convert_ids_to_tokens(ids))
Try the same two sentences with several models' tokenizers, and compute fertility on a hundred Arabic sentences from your own domain, not one. This number goes straight into the cost estimate in the lesson on calling a model.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A token is a word. | A token is a piece that may be a whole word, part of one, or a single letter. In the example «المعلم» became six tokens. |
| Finer cutting means better understanding. | Finer cutting only means a longer sequence and a higher cost. A model does not understand Arabic better because it cuts it into letters. |
| You can swap a trained model's tokenizer for one better at Arabic. | Every id in the vocabulary is tied to a row the model learned in its embedding layer, so changing the tokenizer needs at least that layer retrained. |
Try it yourself
Two models charge the same per million tokens. The first cuts your Arabic sentences at 1.8 tokens per word on average, the second at 3.6. What does that mean?
The price is per token, so twice the tokens is twice the price for the same words, and the same context window holds half the words. Finer cutting does not mean better understanding.
What does BPE merge at each step?
That is all of BPE: count adjacent pairs, merge the top one, repeat. Which is why a tokenizer mirrors exactly what was common in the text it was built from.
Quick review cards
▁), so it suits languages without spaces and the sentence can be rebuilt exactly.Check Your Understanding
Why does a byte-level tokenizer never have an "unknown" token?
Because its vocabulary starts with all 256 possible bytes, and any text in any language or emoji is in the end a sequence of bytes. The worst case is a rare word cut into its single bytes: costly, but nothing is lost.
Why does «النموذج يقرأ النص» have more bytes than characters?
Because an Arabic letter takes two bytes in UTF-8 and a space takes one, so 17 characters became 32 bytes. Diacritics add two bytes per mark, which is why the voweled version jumped to 54.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.