Probability, practically
Lesson 6 of 9
6Probability, practically
The big pictureProbability here is not a separate subject but the tool that stops you trusting a high number in the wrong place, and it explains why high accuracy fools everyone.
What the picture shows: Of every 594 positive results, only 99 belong to people who are actually sick and 495 are false alarms. The test is accurate but the disease is rare, so the many healthy people produce more false alarms than the few sick people produce true hits.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Probability is the language of "how sure are we". If a medical test says you are ill, are you really? It depends on how rare the illness is to begin with. If it is very rare, most positive results may be wrong. This fallacy fools doctors and engineers alike.
A distribution describes possible values and their probabilities, summarised by the mean and standard deviation. Conditional probability P(A|B) changes with information, and Bayes' theorem links it to P(B|A). The base rate fallacy ignores the prior, and it is why classifiers on rare classes get misread.
Before You Start
📊Plots and distributions
You met the histogram in the plots lesson and know data has a shape, not just a mean.
➗Fractions and percentages
You can turn 1% into 0.01 and multiply two proportions without hesitating.
The Core Idea
We take only the probability that shows up in your work: the distribution, then the standard deviation, then conditional probability, then the base rate that fools whoever reads accuracy.
1The distribution: the shape of the data
A distribution describes how likely each value or range of values is. It is the shape of the data, not its middle value.
Look at the picture: a normal distribution with mean 50 and deviation 10. Most values sit near the middle, and they thin out as you move away.
Most of what people call an outlier is a distribution they never looked at. The distribution, not your eye, defines what an outlier is in your data.
2Mean and spread
The mean is a location and the deviation a width, and without the second the first means little. The standard deviation is the average distance of values from their mean.
Look at the picture: how many of a thousand values fall within one deviation and within two.
import numpy as np
rng = np.random.default_rng(7)
sample = rng.normal(loc=50.0, scale=10.0, size=1000)
print(f"mean={float(sample.mean()):.2f}")
print(f"std={float(sample.std(ddof=1)):.2f}")
within_one = float(((sample > 40) & (sample < 60)).mean())
within_two = float(((sample > 30) & (sample < 70)).mean())
print(f"within 1 sd: {within_one:.3f}")
print(f"within 2 sd: {within_two:.3f}")
A thousand values from a normal distribution with mean 50 and deviation 10: the computed mean is 49.28 rather than exactly 50, which is sampling error, not a bug. 0.708 of values fell within one deviation and 0.965 within two, the rule you will use to judge whether a value is genuinely strange.
3Conditional probability
Conditional probability is the chance of something once you know something else. Every classification is one: how likely is this class given these features?
Look at the picture: ten thousand people split twice, once by illness and once by the test result.
If your result is positive, you are in one of the two positive leaves: 99 sick or 495 healthy. The question now is which leaf is bigger.
4The base rate
How common a condition is before any test governs how you read the result. Now compute the two positive leaves, then change only the prevalence and watch the answer.
prevalence = 0.01
sensitivity = 0.99
specificity = 0.95
true_positive = prevalence * sensitivity
false_positive = (1 - prevalence) * (1 - specificity)
positive = true_positive + false_positive
print(f"true positives: {true_positive * 10000:.0f} per 10000")
print(f"false positives: {false_positive * 10000:.0f} per 10000")
print(f"P(sick | positive) = {true_positive / positive:.3f}")
for p in [0.001, 0.01, 0.1, 0.5]:
tp = p * sensitivity
fp = (1 - p) * (1 - specificity)
print(f"prevalence={p:<6} -> {tp / (tp + fp):.3f}")
A test with 99% sensitivity and 95% specificity, for a disease affecting one in a hundred: per ten thousand people there are 99 true positives and 495 false ones, so the chance you are actually ill after a positive result is only 0.167. And look at the final table: the very same test gives 0.952 if the disease affects half the population. The test did not change, the prevalence did.
How It Works, Step by Step: how to compute the chance of illness after a positive result
Start from the base rate: the share who are ill before any test, here 0.01.
Multiply it by the sensitivity to get the true positives: 0.01 × 0.99, which is 99 per 10000.
Multiply the healthy share by the test's error rate on them: 0.99 × 0.05, which is 495 false alarms per 10000.
Divide the true positives by all positive results: 99 out of 594, which is 0.167.
Worked example
99% accuracy as an achievement on a problem whose rare class is 1% of the data. A model that always says "healthy" scores 99% and is completely useless. On imbalanced data look at precision and recall, and the classical machine learning track builds on this in detail.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A 99% accurate test means a positive result is 99% likely to be right. | That depends on how common the condition is. For a disease affecting one in a hundred, a positive result is right only 0.167 of the time. |
| The mean alone describes the data. | The mean is a location and the deviation a width. A class that is half top marks and half failures can share its mean with a class of uniformly average students. |
| A sample mean should equal the true value exactly. | A thousand values from a distribution with mean 50 gave 49.28. The gap is ordinary sampling error, not a bug in the code. |
Try it yourself
Your fraud detector flagged 1000 transactions out of a million and was right about 200 of them. The real fraud rate is 0.05%. What is the most accurate judgement?
Random selection would have been right about 0.05% of the time; the model is right 20% of the time, a four hundred fold improvement. Whether that is useful depends on the cost of reviewing 800 false alerts against the value of stopping 200 real frauds. The fourth option is not entirely wrong, since recall matters, but the question was about reading precision against the base rate.
Quick review cards
99% accurate test for a disease affecting one in a hundred is right about a positive result only one time in six.Check Your Understanding
how do you raise the chance a positive result is correct without improving the test?
By narrowing who you test. Test only people with symptoms or a risk factor and you have raised the base rate within the tested group from one percent to ten or more, so the chance rises from 0.167 to 0.687 as the table shows. Same test, different group, radically different answer. This is exactly what a retrieval stage does before you ask the model.
In a normal sample with mean 50 and deviation 10, is a value of 85 strange?
Yes: it is more than three deviations from the mean, while 0.965 of the values in the lesson fell within just two.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.