Plot before you train
Lesson 4 of 9
4Plot before you train
The big pictureA table of summary numbers can be identical across completely different data. Plotting is the only way to see the difference before the model pays for it.
What the picture shows: The mean is 56.1 and the median 24.5 for one column of ages, and once 340 is excluded the mean returns to 24.6. One value dragged the mean to more than double the median, while the median never moved.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A table of a thousand numbers tells you nothing at a glance. Plot it and you see at once where values cluster, whether one column rises with another, and which point sits far from the rest. Plotting before training stops you building a model on data with an obvious mistake.
A histogram shows a distribution, a scatter plot the relation between two variables, and a box plot the median, quartiles and outliers. You use matplotlib or seaborn and look for skew, impossible values and spurious correlations before any model.
Before You Start
🗂️Frames
You can load a table with pandas and compute a column mean, as in the frames lesson.
📏Mean and median
You know the median is the middle value after sorting, and that it is not the mean.
The Core Idea
This lesson starts with why you plot: summary numbers lie. Then it gives you three plots that are enough to start: the histogram, the scatter, and the box plot with a rule for outliers.
1Summary numbers can match
The mean, the spread and the correlation describe a lot and hide the shape. Try three small datasets and compute the same numbers for each.
import numpy as np
x_a = np.array([10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5], dtype=float)
y_a = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
y_b = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74])
x_c = np.array([8, 8, 8, 8, 8, 8, 8, 19, 8, 8, 8], dtype=float)
y_c = np.array([6.58, 5.76, 7.71, 8.84, 8.47, 7.04, 5.25, 12.50, 5.56, 7.91, 6.89])
for name, x, y in [("A", x_a, y_a), ("B", x_a, y_b), ("C", x_c, y_c)]:
r = float(np.corrcoef(x, y)[0, 1])
print(f"{name}: mean_y={float(y.mean()):.2f} std_y={float(y.std(ddof=1)):.2f} r={r:.2f}")
Three datasets with the same mean, the same spread and nearly the same correlation. Plot them and you would see the first as a tilted cloud, the second as a clear curve, and the third as a single column with one distant point creating the whole correlation by itself. That is the entire argument of this lesson.
2The histogram: how values spread
A histogram answers one question: how are this column's numbers distributed? It counts how many values fall in each range.
Look at the picture: ten ages. Nine sit between 19 and 31, and one stands alone far away at 340.
A histogram also tells you whether a column has two peaks or is skewed, and both need handling before training.
3The scatter: do two columns move together
A scatter puts each row as a point: one column across, the other up. Look at the picture: a tilted cloud with a line through it.
The correlation alone is not enough, because it gave 0.82 to a cloud, a curve and a column with one stray point. Only the plot tells them apart, in a second.
4The box plot and the IQR rule
A box plot packs the median, the quartiles and the outliers into one shape. The box runs from q1 to q3, and the fence sits 1.5 interquartile ranges beyond.
Look at the picture: the ages cut at those limits, and whatever lies past the fence deserves a look.
import numpy as np
ages = np.array([19, 21, 22, 23, 24, 25, 27, 29, 31, 340])
q1, q3 = np.percentile(ages, [25, 75])
iqr = q3 - q1
upper = q3 + 1.5 * iqr
print(f"mean={float(ages.mean()):.1f} median={float(np.median(ages)):.1f}")
print(f"q1={float(q1):.1f} q3={float(q3):.1f} upper_fence={float(upper):.1f}")
print([int(a) for a in ages if a > upper])
kept = ages[ages <= upper]
print(f"mean_after={float(kept.mean()):.1f}")
The mean is 56.1 and the median is 24.5, and that gap alone is an alarm. The rule flagged 340, and once excluded the mean returned to 24.6, close to the median. The median never moved, which is why we always compare the two.
How It Works, Step by Step: how the interquartile rule works
Sort the values and compute the first quartile q1 and the third quartile q3, the two points that hold the middle half of the data.
Compute the interquartile range, iqr = q3 - q1, the width of that middle half.
Set the upper fence at q3 + 1.5 * iqr and the lower one at q1 - 1.5 * iqr.
Every value beyond the fences deserves a look, then you decide: an entry error to fix, or a rare fact to keep.
Worked example
340 is plainly an entry error, but one large transaction in sales data may be the most important row in the table. Inspect the row itself before deciding.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Summary numbers are enough to understand a column. | In the lesson three datasets share the same mean, the same spread and nearly the same correlation, and look nothing alike. Only a plot shows it. |
| Whatever the rule flags must be deleted. | The rule is an attention tool, not a verdict. An age of 340 is a clear error, but one huge transaction may be the most important row in the table. |
| A long tail in income is an error to clean up. | Right skew is normal for income and prices. The usual treatment is a log transform or a robust statistic such as the median, not deleting the tail. |
Try it yourself
An income column has a mean of 18000 and a median of 7000. What is your first conclusion?
A mean far above the median is the working definition of right skew: a long tail of high values drags the mean while leaving the median still. That is entirely normal for income and prices, and the next decision is a log transform or a robust statistic, not deleting the tail for no reason.
Quick review cards
1.5 ranges beyond a quartile deserves a look before any decision.Check Your Understanding
your correlation is 0.9. Is that enough to call the relationship linear?
No. Correlation measures only the strength of a linear relationship, and it can be inflated by one outlier as in the third dataset above, or deflated by a strong but curved relationship. Plot the scatter, because the number alone cannot tell those cases apart.
Which plot do you pick to compare the price distribution across five cities, and why?
A box plot, because it puts each city's median, quartiles and outliers side by side in one shape that is easy to compare.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.