Scaling and load balancing
Lesson 1 of 3
1Scaling and load balancing
The big pictureOne server serves a thousand users. For a million you need many servers, a balancer spreading requests across them, and a design that lets any server handle any request.
What the picture shows: Horizontal scaling: the balancer spreads the load, the servers are identical and replaceable, and state lives in one shared place.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A café with one cashier chokes at peak. You can hire a faster cashier (vertical scaling), but speed has a limit. Or open five tills with someone at the door sending each customer to the shortest queue (horizontal scaling and a load balancer). On one condition: any cashier can serve any customer, meaning orders are recorded in a shared system, not in each cashier's notebook.
Start with a rough estimate: daily active users × requests per user ÷ 86400 = average requests per second, and the peak is usually 2 to 10 times that. Vertical scaling is simple but has a ceiling and a single point of failure. Horizontal scaling needs stateless servers: sessions in Redis or JWTs, files in object storage (S3), never on the server's disk. A balancer (L4 on TCP or L7 on HTTP) uses algorithms: round robin, least connections, weighted, or IP hashing for stickiness when unavoidable. Health checks pull a broken server out, and autoscaling adds servers by CPU or queue length. The database is usually the next bottleneck.
Before You Start
🚢Deployment
The containers and deployment track.
📨HTTP
You know requests and replies.
The Core Idea
1Estimate with numbers
Before any diagram, do the maths. This small function turns business numbers into requests per second and a server count:
import math
def estimate(daily_users, requests_per_user, peak_factor, rps_per_server):
average = daily_users * requests_per_user / 86_400
peak = average * peak_factor
servers = math.ceil(peak / rps_per_server) + 1 # +1 so one can fail
return round(average), round(peak), servers
for users in (10_000, 1_000_000):
avg, peak, servers = estimate(users, requests_per_user=30, peak_factor=5, rps_per_server=200)
print(f"{users:>9,} users: avg {avg} rps, peak {peak} rps, servers {servers}")2Balancing algorithms
Round robin hands out in turn; least connections picks the least busy, which is better when request durations vary. A simulation with some heavy requests:
from itertools import cycle
requests = [5, 1, 1, 5, 1, 1, 5, 1, 1] # seconds of work per request
servers = ["A", "B", "C"]
def round_robin(jobs):
load = dict.fromkeys(servers, 0)
for job, server in zip(jobs, cycle(servers)):
load[server] += job
return load
def least_loaded(jobs):
load = dict.fromkeys(servers, 0)
for job in jobs:
target = min(load, key=load.get)
load[target] += job
return load
print("round robin :", round_robin(requests))
print("least loaded:", least_loaded(requests))3Stateless
If server A keeps the user's session in its memory and the balancer sends the next request to B, the user finds themselves logged out. Move the session to a shared place:
Stateful (breaks when scaled):
server A memory: {session_42: {user: "sara", cart: [...]}}
next request -> server B -> "who are you?"
Stateless (scales):
any server -> Redis: GET session:42 -> {user: "sara", cart: [...]}
uploaded files -> object storage, not local diskWorked example
Problem: a booking app expects 200 thousand daily users, 20 requests each, with a 6× peak. Maths: 200,000 × 20 ÷ 86,400 ≈ 46 rps average and ≈ 278 at peak. One server handles 150 rps, so we need 2 + 1 spare = 3. Result: three stateless servers behind a balancer, not a hundred microservices.
Myth vs Reality
| Myth | Reality |
|---|---|
| Scaling means rewriting everything. | Usually making servers stateless and adding a balancer and a cache is enough. |
| Sticky sessions solve the problem. | They hide it: a server fails and its users lose their sessions, and the balance skews. |
Where you meet it at work
🛍️Sales seasons
Adding servers automatically at peak and removing them after.
🌐High availability
A server fails and the site carries on.
Test your understanding
A million daily users, 10 requests each. The average is roughly?
10,000,000 ÷ 86,400 ≈ 116.
Why must servers be stateless?
State in a shared place makes servers identical.
Requests vary a lot in duration. Which algorithm is better?
It sends to the least busy, so heavy work does not pile on one server.
Quick review cards
Check Your Understanding
Where do sessions live in a scaled system?
In shared Redis, or in a signed token like a JWT on the client.
What separates an L4 from an L7 balancer?
L4 spreads TCP connections without reading them; L7 reads HTTP and routes by path and headers.
After adding servers, what is usually the next bottleneck?
The database, which is the next lesson's topic.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.