Trees, forests and boosting
Lesson 5 of 8
5Trees, forests and boosting
The big pictureOn tabular data, which is most company data, boosted trees still usually beat neural networks. This lesson builds from one tree to a forest to boosting, measuring each step.
What the picture shows: A delivery order passes just two questions on its way to a leaf: further than 15 km, and raining, so the prediction is "late". That is all a decision tree does: ordered yes-or-no questions, each splitting the data into two purer halves.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A decision tree is like the game of twenty questions: is income above this? yes. Is age below that? no. Until you reach an answer. A forest is hundreds of trees voting together so they err less. Boosting is trees in sequence, each fixing the mistakes of the one before.
A tree splits features at thresholds that reduce Gini impurity or entropy, and overfits if left to grow deep. A Random Forest averages trees built on random samples and features. Gradient boosting, XGBoost and LightGBM, builds trees on residuals and is usually strongest on tabular data.
Before You Start
📏Metrics
You know the difference between training and test accuracy, and why the second is the one that counts.
✂️Splitting data
You hold the test set out before any experiment, so every number in this lesson is measured on data the model never saw.
The Core Idea
On tables, boosted trees still usually win. Here you start with one tree and how it memorises, then combine many trees, measure four models, and read what the forest uses.
1Depth controls memorisation
A decision tree asks a chain of questions about a column and a cut-off until it reaches a leaf holding the answer. The more questions you allow, the closer it gets to memorising every row.
Look at the picture: with depth, training climbs to 100%, and test barely moves.
A tree with no depth limit keeps asking until every training row is classified correctly, that is, it memorises. Watch the gap between train and test.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
for depth in [1, 3, None]:
tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_tr, y_tr)
print(f"max_depth={str(depth):4} train={tree.score(X_tr, y_tr):.3f} test={tree.score(X_te, y_te):.3f} leaves={tree.get_n_leaves()}")
No limit: 100% on training and 90.6% on test. The tree memorised every training row with sixteen leaves. The gap between the two numbers is overfitting, and you will see it in every model until the end of the roadmap.
2Many trees: forests and boosting
Different trees make different mistakes. So voting cancels much of the instability and keeps the pattern most of them agree on.
One tree is unstable: change a few rows and it all changes. A random forest trains hundreds of trees, each on a random sample of rows and columns, then takes a vote. Boosting trains small trees in sequence, each correcting the errors of those before it.
Look at the picture: in boosting each tree does not start from scratch. It learns what the ensemble before it missed, then joins it with a small weight.
3Four models on a delivery table
Lateness here comes from cut-offs and interactions: a long distance at rush hour, or rain with a longer distance. Look at the picture: each step from tree to forest to boosting adds accuracy.
Synthetic delivery data: lateness happens if the distance is over 15 km at rush hour, or over 22 km in rain, with 5% noise. The rule is interactions and cut-offs, exactly what a linear model struggles with.
import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
rng = np.random.default_rng(0)
n = 3000
distance_km = rng.uniform(1, 30, n)
hour = rng.integers(0, 24, n)
rain = rng.integers(0, 2, n)
rush = ((hour >= 7) & (hour <= 9)) | ((hour >= 16) & (hour <= 19))
late = ((distance_km > 15) & rush) | ((rain == 1) & (distance_km > 22))
late = late ^ (rng.random(n) < 0.05)
X = np.column_stack([distance_km, hour, rain])
models = {
"logistic": LogisticRegression(max_iter=1000),
"one tree": DecisionTreeClassifier(random_state=0),
"forest": RandomForestClassifier(n_estimators=100, random_state=0),
"boosting": HistGradientBoostingClassifier(random_state=0),
}
print(f"late share: {late.mean():.2f}")
for name, model in models.items():
scores = cross_val_score(model, X, late, cv=5)
print(f"{name:9} {scores.mean():.3f}")
Logistic regression scores 83.2%, barely above the 73% baseline (the not-late share), because hour is not linear: rush hour is morning and evening, not "the later, the later". The tree reaches 89.4%, the forest 93.0%, and boosting 94.8%, very close to the 95% ceiling the noise imposes.
Trees need no input scaling, care little about outliers, and capture interactions on their own. Those are exactly what tables need.
4Which columns the forest uses
A forest tells you which columns it used most. That is useful for exploring, but it is biased toward columns with many distinct values, and it never means cause.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
data = load_breast_cancer()
forest = RandomForestClassifier(n_estimators=200, random_state=0).fit(data.data, data.target)
ranked = sorted(zip(forest.feature_importances_, data.feature_names), reverse=True)[:4]
for score, name in ranked:
print(f"{score:.3f} {name}")
The top four columns all measure tumour size and shape, and three of them (perimeter, radius and area) measure nearly the same thing. A practical warning: when columns are similar they split the importance among themselves, so each can look moderately important even though the concept they share is the most important.
How It Works, Step by Step: how a tree picks its next question
The tree tries every column and every possible cut point, such as "distance > 10" and "distance > 15".
For each cut it measures how pure the two halves are: a half that is all "late" is purer than a mixed one.
It picks the cut that gives the purest halves, and repeats the same inside each half.
It stops at a set depth or when a half becomes too small, and what is in the leaf is the prediction.
Worked example
A decision tree asks yes-or-no questions about the inputs until it reaches an answer, like a game of twenty questions. Each question picks the column and cut-off that best separate the classes.
HistGradientBoostingClassifier with default settings in seconds.
early_stopping. If boosting does not clearly beat the forest, the forest is easier to maintain.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A deeper tree is always more accurate. | Only on training data. With no depth limit a tree memorises every row and falls over on new data. |
| Neural networks beat everything. | On tables, gradient boosting usually wins, and it is faster and easier to tune. |
| Feature importance means the feature is the cause. | It means the tree used it a lot, not that it causes the outcome. Two correlated columns split the importance arbitrarily. |
Try it yourself
A decision tree scores 100% on training and 78% on test. First move?
A large gap means memorisation. Constraining the tree forces more general rules, and a forest reduces instability by voting. Scaling changes nothing for trees since they compare values to cut-offs, and scaling does not change the order.
In which of these problems do you expect a neural network to beat gradient boosting?
Images are unstructured: one pixel means nothing, and meaning lives in spatial patterns across thousands of pixels, which is what convolutional networks are built for. The other three are tables of meaningful columns, where boosting usually wins.
Quick review cards
Check Your Understanding
Why does each tree in a random forest also train on a random subset of columns, not just rows?
Because if one column were very strong, every tree would start with it and they would all look alike, so their errors would match and voting would not help. Hiding some columns from each tree forces it to find different routes, so the errors differ and voting cancels more of them.
Why is the train-test gap smaller in a random forest than in one tree?
Because each tree sees a different sample and different columns, so their errors differ, and the vote cancels much of them.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Articles2
Done with this lesson?
Sign in to save your progress.