Language and transformers

Attention step by step, until you build a small transformer and train it on Arabic text.

TokenisationBPE and SentencePiece, and why an Arabic sentence costs more tokens.
EmbeddingsMeaning as a vector and cosine similarity by hand, and where it misleads.
AttentionQuery, key and value computed on paper for a four word sentence, then in code.
The transformer blockMulti-head attention, layer norm, residuals and the feed-forward, together.
PositionWhy a transformer needs order, and RoPE as today's common answer.
PretrainingNext token prediction, and how ability emerges from so naive an objective.
Build a small transformerFrom zero in PyTorch, a million parameters, ready for Arabic text.
Project: a transformer on an Arabic bookTrain it on one book and generate text. It will be bad, and that is the proof.