Pretraining
Lesson 6 of 8
6Pretraining
The big pictureEvery large language model trains on one naive task: guess the next token. Nobody teaches it grammar, facts or programming. But to guess the next word well across trillions of sentences, the model is forced to learn all of it. This lesson is about the objective, the data and the cost.
What the picture shows: This cycle is all of pretraining: the model reads what came before, guesses the next token, its error is measured against the real token from the text itself, then its weights are adjusted. Nobody writes labels, so any text in the world is ready training examples.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Pretraining is one game the model plays billions of times: we hide the next word and ask it to guess. To win this simple game on the whole world's text, it is forced to learn grammar, facts and ways of reasoning. That is why a large ability comes out of a naive goal.
The goal is next-token prediction with cross-entropy loss on huge unlabelled text (self-supervised). scaling laws tie the loss to model size, data and compute. Instruction tuning and human preference tuning come after, so the model follows instructions.
Before You Start
🔁A training loop by hand
You have written a training loop: loss, then backward, then an optimiser step. Pretraining is the same loop over a huge text.
🎭Attention and the mask
You know the causal mask stops each position from seeing what follows it, which is what makes every position its own example.
The Core Idea
In this lesson you see how text makes its own labels, how to read the loss, the objective as a PyTorch line, and finally what training costs.
1The objective: the next token
Every large language model trains on one task: guess the next token. Nobody needs to label anything: the input is everything before a position, and the target is the token at it.
tokens = ["<s>", "the", "cat", "sat", "on", "the", "mat"]
inputs, targets = tokens[:-1], tokens[1:]
for i in range(len(inputs)):
print(f"given {inputs[:i + 1]!s:45} predict {targets[i]!r}")
In real code we do not build these examples one by one. We take the whole sequence as input, the same sequence shifted by one as target, and the causal mask ensures each position sees only what came before. So a 2,048-token sequence gives 2,048 examples in one pass.
2The loss and its reference
We measure quality with the loss: the average −log of the probability the model gave the right token. Uniform guessing gives ln(V), and any model that learns something goes below it.
The simplest possible language model: it predicts the next letter from the current one only, by counting pairs. We measure it with cross-entropy loss on text it has not seen, and compare with uniform guessing.
import math
from collections import Counter
text = ("العلم نور والجهل ظلام. من طلب العلم سهر الليالي. "
"الكتاب خير جليس في الزمان. القراءة تفتح أبواب المعرفة. "
"من جد وجد ومن زرع حصد. العقل السليم في الجسم السليم. "
"الصبر مفتاح الفرج. خير الكلام ما قل ودل. ")
split = int(len(text) * 0.8)
train, test = text[:split], text[split:]
vocab = sorted(set(text))
V = len(vocab)
pairs = Counter(zip(train, train[1:]))
firsts = Counter(train[:-1])
def prob(a, b):
return (pairs[(a, b)] + 1) / (firsts[a] + V) # add-one smoothing
loss = -sum(math.log(prob(a, b)) for a, b in zip(test, test[1:])) / (len(test) - 1)
print("vocabulary:", V, "characters")
print(f"loss of guessing uniformly: {math.log(V):.2f}")
print(f"loss of the bigram model: {loss:.2f}")
def generate(start, n):
out = start
for _ in range(n):
a = out[-1]
out += max(vocab, key=lambda b: prob(a, b))
return out
print("greedy text:", generate("ال", 30))
The loss fell from 3.33 (uniform guessing over 28 letters) to 2.91. Better, but the generated text loops: "الي الي الي". The model knows "ي" often follows "ل" and nothing about what came before that. The transformer fixes this by looking at everything before instead of one letter.
ln(vocabulary size), the loss of uniform guessing. Start every training run by checking that the first loss is close to that number.3The objective in PyTorch
The shift from the first picture and the loss you just read come together in code as three lines.
Shift by one, then the loss. This line is all of "pretraining" as far as code goes.
import torch.nn.functional as F # tokens: (batch, T + 1) integer ids from the text x, y = tokens[:, :-1], tokens[:, 1:] logits = model(x) # (batch, T, vocab) loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), y.reshape(-1))
reshape(-1, vocab) turns every position in every sequence into an independent classification example over vocab classes. You will write this yourself in the transformer we build in the next lesson.
4The cost: 6 × N × D
The objective is simple, but its cost grows with two things together: the number of parameters N and the number of training tokens D. Double either and the cost doubles.
The rule: each parameter needs about six operations per token (two for the forward pass, four for the backward). We assume a card doing 400 trillion operations per second, running at 40% efficiency.
def train_flops(params, tokens):
return 6 * params * tokens # forward + backward, the usual estimate
gpu_flops = 400e12 * 0.4 # a modern GPU at 40% utilisation
for name, n, d in [("our small model", 1e6, 5e6),
("a 1B model", 1e9, 20e9),
("a 7B model", 7e9, 2e12),
("a 70B model", 70e9, 15e12)]:
f = train_flops(n, d)
seconds = f / gpu_flops
when = f"{seconds:.1f} seconds" if seconds < 3600 else f"{seconds / 3600:,.0f} GPU-hours"
print(f"{name:16} {f:9.1e} FLOPs ~ {when}")
Our small project model takes about a second on a modern card (and minutes on an ordinary CPU). A 70-billion model takes about 11 million card-hours, meaning thousands of cards for months. That gap is why your real work will usually be tuning or calling existing models, not training them from scratch.
That is why few companies pretrain large models from scratch, and everyone else tunes or calls them.
How It Works, Step by Step: how one sentence becomes training examples
Take the whole sequence as input, and the same sequence shifted by one token as the target.
The causal mask lets each position see only what came before it, so every position becomes its own question: what is the next token?
Compute the loss: the average −log of the probability the model gave the right token at each position.
Compare the first loss with ln(V), then repeat the loop over billions of tokens and watch it fall.
Going deeper
Scaling laws found that loss falls in a regular, predictable way as parameters, data and compute grow. The Chinchilla paper found the compute-optimal number of tokens is about 20 times the parameter count. Modern models go far beyond this on purpose, because a smaller model trained longer is cheaper to run afterwards.
Worked example
The beauty of this objective is that the data makes its own labels. Any text in the world is thousands of ready training examples: the input is everything before a position, the target is the token at it.
ln(vocabulary size) for uniform guessing, and a simple bigram model. If your transformer does not clearly beat the bigrams, there is a bug in the code before anything else.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A model after pretraining is a ready assistant. | It only continues text, and may answer your question with more questions. The assistant behaviour comes from tuning on conversations, then on human preferences. |
| Pretraining needs labels written by people. | The next token in the text is the label. This is called self-supervised learning, and it is why training on trillions of tokens became possible. |
| More data always beats cleaner data. | Removing duplicates and junk matters as much as size. Cleaning is a core part of pretraining, not a side step. |
Try it yourself
A model with a 50k-token vocabulary. Its loss at the first training step is 10.8. Is that normal?
A model that has learned nothing gives roughly equal probability to every token, so its loss is ln(V). If it starts much higher, the initial weights are too large.
Where do pretraining labels come from?
This is called self-supervised learning: the text supervises itself. That is why training on trillions of tokens without any human labelling became possible.
Quick review cards
−log of the probability the model gave the right token.6 × parameters × tokens operations. A seven-billion-parameter model on two trillion tokens needs tens of thousands of GPU-hours on modern cards.Check Your Understanding
Why does a pretrained model continue your question with more questions instead of answering?
Because it only learned what usually follows such text, and on the web a list of questions often follows a question. It has not yet learned that its role is "an assistant who answers". That is what tuning on conversations adds. Which is why you pick the Instruct or Chat version of a model when building an application.
Why did the character bigram model repeat "الي الي الي" endlessly?
Because it sees only the current letter and always picks the most likely next one, so it falls into a loop. A transformer breaks it because it looks at everything before, not one letter.
By the 6 × parameters × tokens rule, how much more does training on twice the tokens cost?
Twice: the cost is linear in tokens and in parameters, so doubling either one doubles it.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.