Project: a transformer on an Arabic book
Lesson 8 of 8
8Project: a transformer on an Arabic book
The big pictureIn this project you train the transformer you built on real Arabic text, then make it write. It will write Arabic-looking text with no meaning, and that is exactly the proof that everything you learned works: from random letters to words and sentences with correct spelling, in minutes, from the next token alone.
What the picture shows: Both curves fall together from 3.83 in the first 500 steps, then part after the dashed line at step 2,000: training keeps falling while validation stalls. That gap is the start of memorising, and the sign that what helps now is more text, not longer training.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
You train your small transformer on one Arabic book, then ask it to write. Odd and sometimes funny sentences will come out, but they will echo the book's style and words. That echo is the proof your model really learned, even though it saw only one book.
You prepare and tokenise the text, split it into training and validation, train while logging loss and perplexity, then generate by sampling with temperature and top-k. The report compares the loss to a baseline and shows samples at different temperatures.
Before You Start
🔨Build a small transformer
You have gpt.py with its Block and TinyGPT classes, and it passed the three pre-training checks.
✂️Splitting the data
You know about leakage between training and validation, and why the validation part is held out before any experiment.
The Core Idea
In this project you prepare an Arabic text, train the transformer you built on it and read the loss curve, then make it write at two temperatures.
1Preparing the text
You need a long Arabic text, half a million characters or more. You clean it, map each character to a number, then split it in order, not at random.
We strip diacritics and tatweel, keeping only Arabic letters, the space and three punctuation marks. This keeps the vocabulary small (about 40 characters), so the model learns spelling quickly.
import re
raw = open("raw_book.txt", encoding="utf-8").read()
text = re.sub(r"[\u064B-\u0652\u0640]", "", raw) # diacritics and tatweel
text = re.sub(r"[^\u0621-\u064A \.،؟]", " ", text) # keep Arabic letters and . ، ؟
text = re.sub(r"\s+", " ", text)
open("book.txt", "w", encoding="utf-8").write(text)
print(len(text), "characters,", len(set(text)), "distinct")
Read the first 500 characters of book.txt with your own eyes before any training. If you see broken words or strange characters, fix the cleaning now, because the model will faithfully learn every flaw in it.
2Training and reading the curve
At each step we draw 32 snippets of length 128, the target being the same snippet shifted by one character. We measure training and validation every 500 steps.
If both losses fall together, the model is learning. If training loss keeps falling while validation stalls, it has started memorising.
Put it in train.py next to gpt.py and book.txt, and run it.
import math
import time
import torch
from gpt import TinyGPT
torch.manual_seed(0)
text = open("book.txt", encoding="utf-8").read()
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])
split = int(len(data) * 0.9)
train_data, val_data = data[:split], data[split:]
print(f"characters: {len(text):,} vocabulary: {len(chars)} ln(vocab) = {math.log(len(chars)):.2f}")
ctx, batch = 128, 32
def get_batch(d):
i = torch.randint(len(d) - ctx - 1, (batch,))
x = torch.stack([d[j:j + ctx] for j in i])
y = torch.stack([d[j + 1:j + ctx + 1] for j in i])
return x, y
@torch.no_grad()
def estimate(model):
model.eval()
out = []
for d in (train_data, val_data):
out.append(sum(model(*get_batch(d))[1].item() for _ in range(20)) / 20)
model.train()
return out
model = TinyGPT(len(chars), ctx=ctx)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
start = time.time()
for step in range(3001):
if step % 500 == 0:
tr, va = estimate(model)
print(f"step {step:4} train {tr:.2f} val {va:.2f}")
x, y = get_batch(train_data)
_, loss = model(x, y)
opt.zero_grad()
loss.backward()
opt.step()
print(f"trained in {time.time() - start:.0f}s on CPU")
prompt = torch.tensor([[stoi[c] for c in "البرنامج "]])
for t in (0.5, 1.0):
out = model.generate(prompt, 150, temperature=t)[0]
print(f"\n--- temperature {t}:")
print("".join(chars[i] for i in out))
This code needs PyTorch on your machine, so the output shown is recorded from a real run. Install it with pip install torch and run it yourself.
Read the curve first: the loss started at 3.83, near ln(40) = 3.69, fell fast in the first 500 steps (learning common letters and spaces), then slowly. After step 2,000 validation loss stalled at 1.64 while training loss kept falling to 1.37. That widening gap is the start of memorisation: with this text the model will not improve with longer training, only with more text.
Then the text: compare it with the random letters in the build lesson. Exactly the same architecture, and the difference is minutes of predicting the next character. At temperature 0.5 there are real, repetitive words; at 1.0 more variety and more spelling mistakes.
3Temperature
Temperature divides the scores before softmax. Take three tokens scoring [2, 1, 0]: at 1.0 the first gets 67%, and at 0.5 the scores become [4, 2, 0] and it gets 87%.
Look at the picture: a low temperature piles the probability on the highest-scoring token, so the text becomes repetitive and safe. A temperature of 1.0 spreads it out, so the text is bolder and has more mistakes. That is what you saw in the two generated texts.
How It Works, Step by Step: how the model learns to write from a book
Clean the text, map each of the 40 characters to a number, and hold out the last 10% for validation.
At each step draw 32 snippets of length 128, with the target being the same snippet shifted by one character.
Compute the loss and update the weights with AdamW, measuring training and validation every 500 steps.
At the end generate 150 characters from «البرنامج» at two temperatures, and compare both with the random letters from before training.
Worked example
You need a long Arabic text, half a million characters or more. A book from an open digital library is enough, or a collection of articles. In the recorded run here we used the Arabic text of this platform's software engineering lessons: about 546 thousand characters.
ln(vocabulary size) and below the bigram model from the pretraining lesson. And two generated texts at different temperatures, with a paragraph from you describing the difference and what the model did and did not learn.
layers, or the width, or the number of steps, and note the validation loss for each run in a table. That small table is the first scaling law you discover yourself.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Meaningless text means the model failed. | Correctly spelled words and Arabic structure from a million parameters in minutes is the expected success. Meaning needs thousands of times the parameters and data. |
| Lower training loss always means a better model. | After step 2,000 training fell from 1.46 to 1.37 and validation did not improve. The number that counts is the validation loss. |
| A higher temperature makes the model smarter or more creative. | Temperature does not change what the model learned, only how it samples from it. At 1.0 there was more variety and more spelling mistakes. |
Try it yourself
Why do we split the text in order (first 90%, last 10%) rather than randomly by character?
Every training snippet is 128 characters long. If positions were mixed, the model would see most of every validation example's context during training. It is the same leakage lesson from the machine learning track.
The trained model writes correctly spelled words with no meaning. What is the best explanation?
This is exactly what to expect at this size. Coherent meaning appears with billions of parameters and trillions of tokens, and the same architecture you wrote.
Quick review cards
Check Your Understanding
Training loss is 1.2 and validation loss 1.9 and rising. What is happening, and what do you do?
The model is memorising the training text instead of learning general patterns. The fixes, in order: more text (the best), or stop at the lowest validation loss, or add dropout, or shrink the model. More text almost always wins.
The loss started at 3.83, and ln(40) is 3.69. Is that worrying?
No, the gap is small and expected because random initial weights are not perfectly even. What would worry you is starting far away, such as 20.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.