Project: a network from zero
Lesson 8 of 8
8Project: a network from zero
The big pictureThis track's project: a network you write from the first line to the last, with a hand-written training loop and a validation curve you read, then three faults we plant in the code for you to find from their symptoms before reading the answer.
What the picture shows: The three faults can be told apart from just two numbers: accuracy and loss. The lit path reaches the most dangerous one, the extra softmax: 93% accuracy looks healthy, but a loss of 0.408 instead of 0.284 gives it away.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
In this project you write the whole training loop by hand with no ready-made tool, then we hide three bugs in the code, and you find them as a mechanic finds a car fault: by observation and testing, not guessing.
You build a small model, a Dataset, a DataLoader and a training and evaluation loop with logged curves. The planted faults are the forgotten zero_grad kind, shifted labels and a missing model.eval(), and you diagnose each from the curve or by overfitting one batch.
Before You Start
⚙️A training loop by hand
You can write the five lines without looking, with model.train(), model.eval() and torch.no_grad() in their places.
🩺Debugging training
You read training and validation curves together, and know that ln(number of classes) is the loss of a model guessing evenly.
The Core Idea
This track's project is a network you write from the first line. You build the network on the two moons data, then train it and read its curve, then keep the best model, then plant three faults and recognise them from their numbers.
1The data and the network
The two moons data is points on a plane in two interlocking shapes no straight line can separate. We generate 1000 points, 800 for training and 200 for validation, and fix the seed.
Look at the picture: two inputs, two hidden layers of 32 units each followed by ReLU, and two outputs. The network is small on purpose, so training takes seconds.
2The loop and the curve
The loop is the five lines, with model.train() before training and model.eval() and torch.no_grad() before evaluating. Look at the picture: validation loss falls until epoch 10, then rises while training loss keeps falling.
This code needs PyTorch on your machine, so the output shown is recorded from a real run. Install it with pip install torch and run it yourself.
import torch
from sklearn.datasets import make_moons
from torch import nn
X, y = make_moons(n_samples=1000, noise=0.25, random_state=0)
X = torch.tensor(X, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.long)
X_train, y_train, X_val, y_val = X[:800], y[:800], X[800:], y[800:]
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 2))
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(1, 201):
model.train()
for i in range(0, 800, 64):
xb, yb = X_train[i:i + 64], y_train[i:i + 64]
loss = loss_fn(model(xb), yb)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if epoch in (1, 10, 50, 200):
model.eval()
with torch.no_grad():
train_loss = loss_fn(model(X_train), y_train).item()
val_logits = model(X_val)
val_loss = loss_fn(val_logits, y_val).item()
val_acc = (val_logits.argmax(dim=1) == y_val).float().mean().item()
print(f"epoch {epoch:3} train {train_loss:.3f} val {val_loss:.3f} val acc {val_acc:.3f}")
After ten epochs: 93.5% validation accuracy and a validation loss of 0.206, the best model of the run. After that training loss creeps down while validation loss climbs to 0.314, with accuracy flat at 93%. Accuracy did not move, but the model grew more confident in its mistakes, which the loss reveals and accuracy does not.
3Keep the best model
The model you want is the one at the lowest validation loss, not the one at the last epoch. That is early stopping in its simplest form.
Early stopping in its simplest form: track the lowest validation loss, save the weights there, and load them at the end.
best_loss, best_state, patience, waited = float("inf"), None, 10, 0
for epoch in range(200):
train_one_epoch(model, train_loader)
val_loss = evaluate(model, val_loader)
if val_loss < best_loss:
best_loss, waited = val_loss, 0
best_state = {k: v.clone() for k, v in model.state_dict().items()}
else:
waited += 1
if waited >= patience:
break
model.load_state_dict(best_state)
patience gives the model ten epochs to improve before stopping, since validation loss wobbles rather than climbing in a straight line. clone() is essential: without it you store references to the current weights, which will change, not a copy.
4Three planted faults
Copy the file three times and plant one fault in each copy: zero_grad removed, an extra softmax, and randomly shuffled labels. Look at the picture: the last loss of each copy, and the marked one is the most dangerous.
Before reading the explanation, guess from the numbers: which fault produces which line?
import torch
from sklearn.datasets import make_moons
from torch import nn
X, y = make_moons(n_samples=1000, noise=0.25, random_state=0)
X = torch.tensor(X, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.long)
X_train, y_train, X_val, y_val = X[:800], y[:800], X[800:], y[800:]
loss_fn = nn.CrossEntropyLoss()
def run(bug):
torch.manual_seed(0)
layers = [nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 2)]
if bug == "softmax":
layers.append(nn.Softmax(dim=1))
model = nn.Sequential(*layers)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
for epoch in range(10):
for i in range(0, 800, 64):
xb, yb = X_train[i:i + 64], y_train[i:i + 64]
if bug == "labels":
yb = y_train[torch.randperm(800)[:len(xb)]]
loss = loss_fn(model(xb), yb)
if bug != "zero_grad":
optimizer.zero_grad()
loss.backward()
optimizer.step()
with torch.no_grad():
acc = (model(X_val).argmax(dim=1) == y_val).float().mean().item()
return loss.item(), acc
for bug in ["none", "zero_grad", "softmax", "labels"]:
loss, acc = run(bug)
print(f"{bug:9} last loss {loss:.3f} val acc {acc:.3f}")
zero_grad removed: a loss of 4.03, far above the start, and 53.5% accuracy, about chance. Extra softmax: 93% accuracy, nearly the healthy one, but a loss of 0.408 versus 0.284, because gradients got weaker and learning slower. Shuffled labels: a loss of 0.697, almost exactly ln(2) = 0.693, and 50.5% accuracy. That specific number, ln of the class count, is the sign that the model is guessing evenly.
The goal is a visual memory: when you see these signatures in a real project months from now, you know where to look straight away.
How It Works, Step by Step: how to build the project and plant its faults
Generate 1000 two-moons points, 800 for training and 200 for validation, and fix the seed.
Train a network of three Linear layers with ReLU between them using a hand-written loop, printing training loss, validation loss and validation accuracy.
Keep the weights at the lowest validation loss by copying state_dict with clone(), and plot both curves to an image file.
Copy the file three times, plant one fault in each copy, then log its loss, accuracy and symptom in a table.
Worked example
The data: scikit-learn's "two moons", points on a plane in two interlocking shapes no straight line can separate. Easy to plot, and enough to show every phenomenon you met in this track.
model.eval() with Dropout, or a learning rate of 10, and add it to the table. The bigger your table, the faster your diagnosis.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| If accuracy did not change, the model did not change. | Accuracy stayed at 93% from epoch 10 to 200 while validation loss rose from 0.206 to 0.314: the model grew more confident in its mistakes. |
| The dangerous faults are the ones that raise errors. | A fault that fails loudly gets fixed in minutes. The dangerous one is silent, like the extra softmax that gives accuracy that looks fine. |
| Saving the best model only needs best_state = model.state_dict(). | That stores references to the current weights, which will change. You need a copy made with clone() for each tensor. |
Try it yourself
A ten-class network's loss is stuck at 2.30 from the first epoch to the last. Most likely?
ln(10) ≈ 2.30 is the loss of a model giving 10% to each class, guessing evenly. Staying exactly there means it learned nothing at all, and even a small model would have learned something. Check the data, the labels and gradient flow with the one-batch test.
Why is the extra-softmax fault more dangerous in practice than the zero_grad one?
A 93% accuracy looks like success, so the code ships with a model that learns slower and is worse calibrated than it could be. A fault that fails loudly gets fixed in minutes; a silent one lives for months.
Quick review cards
zero_gradCrossEntropyLossln(number of classes) and chance accuracy. The model finds no relationship because there is none.Check Your Understanding
In the healthy run accuracy stayed at 93% from epoch 10 to 200 while validation loss rose. Which model do you pick, and why?
The epoch-10 model, with the lowest validation loss. Accuracy is equal, but the rising loss means the later model gives more extreme probabilities to its mistakes, so if you later use its probability as a confidence score or to pick a threshold, it will mislead you. The earlier model also memorises less, so it generalises better to data a little different from validation.
Why does patience = 10 give the model ten epochs before stopping instead of stopping at the first rise in validation loss?
Because validation loss wobbles rather than climbing in a straight line, so one rise may be followed by a new low. Patience stops you quitting too early over passing noise.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Done with this lesson?
Sign in to save your progress.