Derivatives and gradients
Lesson 8 of 9
8Derivatives and gradients
The big pictureEvery training run in the rest of this roadmap repeats one sentence: compute the loss, compute its gradient, step against it, repeat. Here you write that sentence by hand before any framework.
What the picture shows: Both paths start at w = 0 and head for the bottom at 3, and the steps shrink on their own as they get close because the gradient shrinks. Rate 0.3 almost arrives within a few steps, while rate 0.1 is still at 2.7938 after twelve.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Imagine standing on a hill in thick fog wanting to reach the valley. You cannot see the path, but you feel the slope under your feet, take a small step downhill, then feel again. That is exactly what a model does when it learns: measure the slope and step against it until the error is lowest.
A derivative measures how a function changes for a small change in its input; the gradient is the vector of partial derivatives pointing to steepest ascent. gradient descent updates weights w = w - lr * grad, and the learning rate lr sets the step length: too large oscillates, too small crawls.
Before You Start
📈Slope of a line
You know a slope says how much a line rises when you move one step to the right.
🐍Functions and loops
You can write a Python function and a for loop that updates a variable on each pass.
🧭Vectors
You know from the linear algebra lesson that a vector is a list of numbers, so a gradient in many dimensions is a vector.
The Core Idea
This lesson builds the whole sentence of training by hand: the gradient as a slope at a point, then the step against it, then the step length, then the gradient when there are many weights.
1The derivative: a slope at a point
Take a simple loss: (w - 3) ** 2. Look at the picture: its value at w from 0 to 6, with the bottom at 3.
The derivative says: if you nudge w a little, how much does the loss change? Its sign tells you which way is up. At w = 0 the gradient is -6, so the loss falls as w grows.
2Stepping against the gradient
The gradient points uphill and we want to go down, so we take exactly the opposite direction. Look at the picture: the sentence every training run repeats.
With rate 0.3 you start at w = 0, the gradient is -6, so w becomes 1.8. Then the gradient is -2.4, then -0.96, so the steps shorten on their own as you near the bottom.
That is why training slows near the end, and it is not a defect.
3The learning rate: step length
The learning rate multiplies the gradient before the step, so it sets the step length. Look at the picture: the distance from the bottom at 3, for two rates.
Rate 0.1 closes in slowly and is still at 2.7938 after twelve steps. Rate 1.1 heads the right way with a step longer than the distance, so it overshoots until it reaches -23.7 with a loss of 715.
4The gradient in many dimensions
A network has millions of weights, so the gradient is a vector of the same length: one partial derivative per weight. Look at the picture: the forward pass computes the loss, then the gradient flows back to every layer.
Then every weight is updated in one step by the same rule you applied to w alone. Building that vector automatically is what libraries call autograd.
How It Works, Step by Step: how one gradient descent step works
Compute the loss at the current weight, such as (w - 3.0) ** 2 at w = 0, which is 9.
Compute the gradient at the same point: 2.0 * (w - 3.0) gives -6, and its negative sign says the loss falls if you increase w.
Multiply the gradient by the learning rate and subtract it from w: with rate 0.3, w = 0 - 0.3 × (-6) = 1.8.
Repeat from the new position. The gradient is now -2.4, so the next step is shorter than the last.
Worked example
Example: five steps by hand
def gradient(w):
return 2.0 * (w - 3.0)
w = 0.0
rate = 0.3
for step in range(1, 6):
g = gradient(w)
w = w - rate * g
print(f"step {step}: gradient={g:+.3f} w={w:.3f}")
We started at w = 0 with the bottom at 3. Notice the gradient shrinking each step: -6.000, then -2.400, then -0.960. The closer we get, the flatter the surface, so the steps shorten on their own with no intervention from you. That is why training slows near the end, and it is not a defect.
Example: three learning rates
def loss(w):
return (w - 3.0) ** 2
def gradient(w):
return 2.0 * (w - 3.0)
for rate in [0.1, 0.5, 1.1]:
w = 0.0
for _ in range(12):
w = w - rate * gradient(w)
print(f"rate={rate:<4} w={w:.4f} loss={loss(w):.4f}")
Three identical runs differing only in step length. 0.1 reached 2.7938, still short after twelve steps; 0.5 landed exactly at the bottom; 1.1 exploded to -23.7 with a loss of 715. The third rate was not slower, it was heading the right way with a step longer than the distance to the bottom, overshooting further every time.
NaN and concluding the data is corrupt. Usually the learning rate is far too large and values exploded as in the third run. Divide the rate by ten and rerun before anything else.
NaN diagnoses itself in seconds, while silent training costs you hours.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A NaN loss means the data is corrupt. | Usually the learning rate is far too large and values exploded, as with rate 1.1, where the loss reached 715. Divide the rate by ten first. |
| A larger learning rate always means faster training. | Only up to a point. Once the step is longer than the distance to the bottom, the model overshoots and moves further away each time. |
| Local minima are the biggest problem in training networks. | In practice they matter less than people say; flat plateaus, where the gradient is near zero, are what really slow training. |
Try it yourself
You are training and the loss falls, rises, falls again, oscillating clearly around a fixed value. What do you try first?
Oscillation around a fixed value is the classic sign of too long a step: the model jumps back and forth over the bottom without settling. Lowering the rate or using a schedule that decays it fixes this. More epochs repeat the same oscillation, and layers and data address entirely different problems.
Quick review cards
Check Your Understanding
your loss surface is a long near-flat plateau followed by a steep drop. What do you expect from a small constant rate?
Tedious progress across the plateau, because the gradient there is near zero so each step is tiny, then a violent jump at the drop where the gradient suddenly grows. This is exactly why learning rate schedules and gradient clipping exist, and you will meet both by name in the deep learning track.
Why did rate 0.5 land exactly on the bottom in a single step for this function?
Because the gradient is 2 * (w - 3), so a step of 0.5 × 2 × (w - 3) equals exactly the distance to the bottom. That is special to this simple function and does not happen in real networks.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.