Autograd
Lesson 2 of 8
2Autograd
The big pictureIn the gradient lesson you computed the gradient from a formula you wrote by hand. With millions of weights that is impossible. Autograd records every operation during the forward pass, then walks the record backward with the chain rule and hands you every weight's gradient.
What the picture shows: The computation runs forward and keeps every intermediate value, then walks back from the loss to w. On the way back the local derivatives multiply: 2 × (−3) × 1 × 3, giving the gradient −18 without writing any formula.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Every calculation the model does is recorded like a recipe: "multiplied this by that, then added". When you want to know which ingredient to adjust to improve the result, PyTorch reads the recipe backwards and works out each ingredient's effect automatically. That way you never compute derivatives by hand.
requires_grad=True makes a tensor tracked in a computational graph. loss.backward() applies the chain rule from the loss to the leaves and stores gradients in .grad. Gradients accumulate, so you clear them with zero_grad() every step, and torch.no_grad() stops tracking during evaluation.
Before You Start
📉Derivatives and gradients
You computed a gradient from a formula you wrote by hand, and you know its sign says which way to move the weight.
🧊Tensors
You know a tensor can remember the operations that made it, which is exactly what autograd uses.
The Core Idea
With millions of weights you cannot write the gradient formula by hand. In this lesson you compute one gradient with the chain rule, then let PyTorch compute it, then see why gradients accumulate, then when tracking stops.
1The chain rule by hand
The computation: y = w·x + 1, then loss = (y − 10)². Each forward step has a simple local derivative. The final gradient is their product, from the end back to the start.
w, x, target = 2.0, 3.0, 10.0
# forward, keeping every intermediate value
y = w * x + 1
diff = y - target
loss = diff ** 2
# backward, chain rule from the end to the start
dloss_ddiff = 2 * diff
ddiff_dy = 1.0
dy_dw = x
dloss_dw = dloss_ddiff * ddiff_dy * dy_dw
print(f"loss={loss} dloss/dw={dloss_dw}")
Each forward step has a simple local derivative. The final gradient is their product from the end back to the start: 2 × (7 − 10) × 1 × 3 = −18. The negative sign says: increase w to reduce the loss, which makes sense since y = 7 is below the target of 10.
That is all backpropagation does: it multiplies the local derivatives along the path between the loss and the weight. It writes no formula, and it does not nudge the weight to measure the change.
2The computation graph and requires_grad
When you set requires_grad=True on a weight, PyTorch starts recording every operation that touches it. Each result carries a grad_fn pointing at the operation that made it. Look at the picture: these are the grad_fn names of the three results, and following the chain back from the loss leads you to the weight.
This code needs PyTorch on your machine, so the output shown is recorded from a real run. Install it with pip install torch and run it yourself.
import torch
w = torch.tensor(2.0, requires_grad=True)
x = torch.tensor(3.0)
y = w * x + 1
loss = (y - 10) ** 2
loss.backward()
print("y =", y.item(), "loss =", loss.item())
print("dloss/dw =", w.grad.item())
print("by hand =", 2 * (y.item() - 10) * x.item())
requires_grad=True says "track everything that happens to this tensor". loss.backward() walks backward and stores the gradient in w.grad. The same −18, without writing any formula.
Weights inside an nn.Module have requires_grad on automatically, and data does not need it. The graph is rebuilt on every run, which is why ifs and loops in your code work without trouble.
3Gradients accumulate
backward does not replace .grad, it adds to it. That is useful for accumulating small batches, and destructive if you forget to zero it. Look at the picture: the same loss three times, and the gradient grows each time.
The same loss three times at the same weight. The true gradient each time is −8.
import torch
w = torch.tensor(1.0, requires_grad=True)
for step in range(3):
loss = (w * 2 - 4) ** 2
loss.backward()
print(f"step {step}: grad = {w.grad.item()}")
w.grad.zero_()
loss = (w * 2 - 4) ** 2
loss.backward()
print("after zero_:", w.grad.item())
−8, then −16, then −24: each backward added −8 to what was there. In a training loop every update would be built on the sum of all past gradients, and the weights would shoot off. After zero_() the correct −8 returns.
4Turning tracking off: no_grad and detach
For evaluation and prediction you need no gradient. torch.no_grad() stops building the graph, which saves a lot of memory and time.
import torch
w = torch.tensor([1.0, -2.0], requires_grad=True)
x = torch.tensor([3.0, 4.0])
y = (w * x).sum()
print(type(y.grad_fn).__name__)
print(y.requires_grad)
with torch.no_grad():
z = (w * x).sum()
print(z.requires_grad)
print(w.detach().numpy())
y knows it came from a sum and needs a gradient. Inside torch.no_grad() no graph is built, so z is a plain tensor. detach() gives a copy cut off from the graph, the right way to turn a weight into NumPy for printing or plotting.
How It Works, Step by Step: how PyTorch computes a gradient by itself
You set requires_grad=True on the weight, so PyTorch starts recording every operation that touches it.
During the forward pass every result carries a grad_fn pointing at the operation that made it, which builds the computation graph.
On loss.backward() it walks the graph back from the loss and multiplies the local derivatives by the chain rule.
It adds the gradient to each weight's .grad and then frees the graph, which is why you zero .grad before the next step.
Worked example
Before PyTorch, compute one gradient by the chain rule by hand. The computation: y = w·x + 1, then loss = (y − 10)². We want dloss/dw.
total += loss instead of total += loss.item(). The first keeps every batch's computation graph in memory, and you run out of memory after a few hundred steps without knowing why.
1e-4 and divide the change in loss by 1e-4. If it is close to what backward gives, the code is right. PyTorch ships this as torch.autograd.gradcheck.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Autograd writes out the derivative formula and then evaluates it. | It writes no formula, and it does not nudge weights to measure the change. It applies the chain rule mechanically to the operations that actually ran. |
| backward replaces the old gradient with the new one. | It adds to it. The same loss three times gave −8, then −16, then −24, and only after zeroing does −8 come back. |
| Ifs and loops in your code break autograd. | The graph is rebuilt on every run from the operations that executed, so it works with ifs and loops without trouble. |
Try it yourself
loss = (3 * w - 6) ** 2 at w = 1. What is w.grad after loss.backward()?
The chain rule: 2 × (3·1 − 6) × 3 = 2 × (−3) × 3 = −18. The outer derivative is 2(·) and the inner one is the coefficient of w, which is 3.
You ran loss.backward() twice on the same loss and got "Trying to backward through the graph a second time". Why?
After backward the saved intermediate values are deleted. In normal training that is what you want, since each step builds a fresh graph. If you really need two calls, use retain_graph=True, but it is rarely the right fix.
Quick review cards
requires_grad doesnn.Module have it on automatically; data does not need it.backward doesrequires_grad and adds it to .grad, then frees the graph.backward adds to .grad instead of replacing it. Useful for accumulating small batches, destructive if you forget to zero it every step.Check Your Understanding
Why wrap the evaluation loop in torch.no_grad() when results are correct without it?
Because without it PyTorch builds a graph for every evaluation batch and keeps intermediate values ready for a backward that never comes. That multiplies memory use and slows computation. The results are correct, but you may be unable to evaluate a large batch that would have fitted otherwise.
Why use w.detach().numpy() instead of w.numpy() to turn a weight into NumPy?
Because the weight is attached to the computation graph and requires a gradient. detach() gives a copy cut off from the graph, the right way to print or plot it.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.