Clustering and dimensionality
Lesson 7 of 8
7Clustering and dimensionality
The big pictureClustering and dimensionality reduction work with no right answer, which makes them the easiest tools for fooling yourself. k-means will hand you groups even when none exist, and PCA will hand you axes even when they mean nothing. The question is always: discovery or pretty illusion?
What the picture shows: Scattered points gather around three centres, which is what k-means does: it assigns each point to the nearest centre, then moves each centre to the mean of its points. But it will hand you groups even on random noise, so a pretty picture alone is no proof.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
You have a pile of unlabelled photos and want to group the similar ones. Clustering puts each photo in the nearest group, then moves each group's centre, and repeats. Dimensionality reduction is like photographing a sculpture from its best angle: you lose some detail but see the shape clearly.
k-means picks k centres and alternates assignment and update until stable; k is chosen with the elbow curve or the silhouette score. PCA projects data onto the directions of greatest variance. Clusters can be artefacts of scaling or the algorithm rather than real structure, so you check them.
Before You Start
🔧Scaling
You know from the feature engineering lesson that distance-based algorithms need scaled inputs.
🧭Vectors and distance
You see each row as a point in space, as in the linear algebra lesson, and know what the distance between two points means.
The Core Idea
Here there is no right answer, which makes fooling yourself easy. You run k-means on real groups, choose k with two measures, run it on noise to see the illusion, then reduce dimensions with PCA.
1k-means on real groups
k-means looks for k centres so every point is close to its nearest one. It assumes round groups of similar size.
We make three groups ourselves, then see whether it finds them.
import numpy as np
from sklearn.cluster import KMeans
rng = np.random.default_rng(0)
centres = np.array([[2, 2], [8, 3], [5, 8]])
points = np.vstack([c + rng.normal(scale=0.8, size=(50, 2)) for c in centres])
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(points)
for c in sorted(km.cluster_centers_.round(1).tolist()):
print("centre:", c)
print("sizes:", sorted(np.bincount(km.labels_).tolist()))
We made three groups around (2,2), (8,3) and (5,8), and k-means found centres very close to them, with fifty points each. Here the answer is known because we made the data. In reality you will not know it, and that is the whole challenge.
The result depends on the random starting centres, so n_init=10 reruns it from ten starts and keeps the best. Always scale inputs first, because it relies on distance.
2Choosing k with two measures
Inertia is the sum of squared distances to the centres, and it always falls as k grows. Silhouette runs from -1 to 1 and measures how much closer each point is to its own group than to the next.
Look at the picture: inertia collapses at k = 3 and then flattens, which is the elbow. Silhouette peaks at the same point.
import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
rng = np.random.default_rng(0)
centres = np.array([[2, 2], [8, 3], [5, 8]])
points = np.vstack([c + rng.normal(scale=0.8, size=(50, 2)) for c in centres])
for k in range(2, 7):
km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(points)
print(f"k={k} inertia={km.inertia_:7.1f} silhouette={silhouette_score(points, km.labels_):.3f}")
Inertia drops from 1080 to 193 between k=2 and k=3, then declines slowly: that is the elbow. Silhouette peaks at 0.750 at k=3 and then falls. The two agree, which is rare with real data, where the elbow is usually a smooth curve with no clear corner.
3Groups out of nothing
k-means always gives you k groups, even on pure random noise. It never tells you "there are no groups here".
Points spread uniformly at random over a square, with no groups at all. Ask k-means for four.
import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
rng = np.random.default_rng(0)
noise = rng.uniform(0, 10, size=(150, 2))
km = KMeans(n_clusters=4, n_init=10, random_state=0).fit(noise)
print("sizes:", sorted(np.bincount(km.labels_).tolist()))
print(f"silhouette: {silhouette_score(noise, km.labels_):.3f}")
Four sensibly sized groups and a silhouette of 0.396, which might look acceptable. Plot them in four colours and they would look like very convincing "customer segments". That is why you always compare against noise: if the silhouette on your data is close to what random data of the same shape gives, there are no real groups.
A silhouette above 0.5 is clear separation and around 0.25 is weak. But the real reference is what noise of the same shape gives: 0.750 on our data against 0.396 on noise.
4PCA: how many dimensions you need
PCA rotates the axes and orders them by variance. The first is the direction the data spreads most, and the second is perpendicular to it.
Look at the picture: how many of the 64 components you need to keep a share of the information.
scikit-learn's handwritten digit images: each is 8×8, so 64 columns.
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True)
pca = PCA().fit(StandardScaler().fit_transform(X))
cum = pca.explained_variance_ratio_.cumsum()
print("original dims:", X.shape[1])
for target in [0.5, 0.8, 0.95]:
print(f"{int(target * 100)}% variance -> {int((cum < target).sum()) + 1} components")
Eight components carry half the information and forty carry 95%, so roughly a quarter of the columns are redundant. But remember: "variance" is not "usefulness". A low-variance direction may be exactly what separates the two classes you care about.
How It Works, Step by Step: how k-means finds its groups
Place k centres at random positions.
Assign each point to its nearest centre.
Move each centre to the mean of the points assigned to it.
Repeat the two steps until the centres settle, and rerun it all from several starts with n_init, keeping the best.
Worked example
k-means looks for k centres so every point is close to its nearest one. The algorithm is simple: place random centres, assign each point to the nearest, move each centre to the mean of its points, repeat until stable.
2. n_init guards against a bad start
The result depends on the random initial centres. n_init=10 reruns the algorithm from ten different starts and keeps the best. Always scale inputs before k-means, since it relies on distance.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| If k-means returns groups, the groups exist. | On uniformly random points it returned four sensibly sized groups and a silhouette of 0.396. Always compare against what noise gives. |
| The highest variance in PCA means the most important direction. | Variance is not usefulness. A low-variance direction may be exactly what separates the two classes you care about. |
| You can name a group right after the first run. | Re-cluster on another sample or with another seed first. If the groups shift a lot, a name like "loyal customers" is a story you invented. |
Try it yourself
You ran k-means on customer data without scaling, with "annual spend" in thousands and "visits" from 1 to 30. What do you expect?
Euclidean distance is dominated by the feature with larger numbers. A thousand-riyal gap counts as much as a thousand visits, so visits become nearly invisible. Scale features before any distance-based algorithm.
PCA shows the first two components explain 92% of the variance. What does this mean?
Explained variance says how much of the spread the two components capture, so plotting them is faithful to the data's shape. It does not say the remaining 8% is useless for classification, and it has nothing to do with the number of clusters.
Quick review cards
Check Your Understanding
Why does inertia always fall as k grows, and why does that make it useless on its own for choosing k?
Because an extra centre can never increase distances: at worst it sits unused, and usually it brings some points closer. At k equal to the number of points inertia hits zero, every point its own group. So the lowest inertia always means the biggest k, which is why we look for where large improvement stops, and lean on silhouette or on what the groups mean.
Why does k-means fail on elongated or nested groups?
Because it assigns each point to the nearest centre, it assumes round groups of similar size. An elongated group gets cut into pieces and nested ones get merged by mistake.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.