Project: a service that works
Lesson 6 of 6
6Project: a service that works
The big pictureIn this project you run your model behind a real endpoint, with a numbers panel showing requests, errors and latency, then push real load at it until you know where it breaks. The final report says in numbers: how many requests a second the service handles at acceptable latency, what a thousand requests cost, and what happens beyond that.
What the picture shows: Four workers, each request about 200 ms. Up to half capacity p95 barely moves, then it doubles at 90% and nears three seconds at 98%. That is why you plan for the service to run at 60% to 70% at peak.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
You run your model as a real service: an API taking requests, a dashboard showing latency and errors, then you fire hundreds of requests at it at once to see how it holds up under load. The report says what actually happened in numbers, not what you expect to happen.
A containerised FastAPI service with a health endpoint, Prometheus metrics and a Grafana dashboard, load-tested with a tool like locust or k6. The report gives p50, p95, throughput, error rate and the breaking point.
Before You Start
📦Containers and deployment
Your service is in a container with a version tag and settings from the environment.
📡Monitoring and drift
You have request, error and latency metrics ready for a panel.
The Core Idea
In this project you run your model behind a real endpoint and push load at it until you know where it breaks. You start by computing capacity, watch the queue approach it, design the load test, then write the report.
1Theoretical capacity
Capacity is the number of workers times what each worker serves per second. A worker taking 200 ms per request serves 5 requests a second.
Look at the picture: four workers give a capacity of 20 requests a second.
2The queue near capacity
Theoretical capacity is not the load you run at. The closer you get, the more requests wait their turn, and latency rises with no extra compute.
import heapq
import random
def simulate(rate_per_s, workers=4, service_ms=200, n=5000, seed=0):
rnd = random.Random(seed)
t, free_at, waits = 0.0, [0.0] * workers, []
heapq.heapify(free_at)
for _ in range(n):
t += rnd.expovariate(rate_per_s) # next request arrives
start = max(t, heapq.heappop(free_at)) # wait for a free worker
service = rnd.expovariate(1 / (service_ms / 1000)) # how long this one takes
heapq.heappush(free_at, start + service)
waits.append((start + service - t) * 1000)
waits.sort()
return waits[n // 2], waits[int(n * 0.95)]
capacity = 4 / 0.2 # 4 workers x 5 requests/s each
print(f"capacity: {capacity:.0f} requests/s")
for rate in [5, 10, 15, 18, 19.5]:
p50, p95 = simulate(rate)
print(f"{rate:5} req/s ({rate / capacity:4.0%} busy): p50 {p50:6.0f} ms p95 {p95:7.0f} ms")
Up to 50% of capacity, latency is nearly flat. At 75% it starts rising, at 90% p95 has doubled, and at 98% it nears three seconds for a request that truly takes 200 ms. All that extra time is queueing, not compute. That is why you plan capacity for the service to run at 60% to 70% at peak, not 95%.
3The load test
A good load test resembles real use: inputs from the real length distribution, load rising in stages, and from a machine other than the server.
Look at the picture: four stages of two minutes each, with the user count doubling.
It needs pip install locust and your service running, so it is here for reading.
import json
import random
from locust import HttpUser, task, between, LoadTestShape
SAMPLES = [json.loads(line)["text"] for line in open("real_inputs.jsonl", encoding="utf-8")]
class ApiUser(HttpUser):
wait_time = between(0.5, 1.5)
@task
def predict(self):
self.client.post("/predict", json={"text": random.choice(SAMPLES)})
class Stages(LoadTestShape):
stages = [(120, 5), (240, 10), (360, 20), (480, 40)] # (end second, users)
def tick(self):
t = self.get_run_time()
for end, users in self.stages:
if t < end:
return users, users
return None
Inputs come from a file of real (redacted) messages, and load rises over four stages of two minutes each. Run it with locust -f load.py --host http://server:8000 --headless --csv results and record for each stage: requests per second, p50 and p95, and the error rate, from both the panel and the results file.
4The panel and the report
During the test, watch the panel, not just the load tool: requests per second, error rate, and p50 and p95 per version. The panel is what will tell you about problems in production.
Look at the picture: the report is one page in four parts.
How It Works, Step by Step: how to run the load test
Write the p95 target before the test, such as "under 500 ms".
Prepare real, redacted inputs with the real length distribution, and run the load tool from a machine other than the server.
Raise the load in stages, two minutes each, and record requests per second, p50, p95 and errors for each stage.
Capacity is the last stage before the target is exceeded or errors appear, and from it you work out the cost per thousand requests.
Worked example
Before any load test, understand why latency collapses near maximum capacity. A service with four workers, each request taking 200 ms on average, has a theoretical capacity of 20 requests a second. Let us see what happens to latency as we approach it.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A server with capacity for 20 requests a second is enough for a peak of 18. | At 90% p95 doubles, any small rise tips it into collapse, and one server means any failure stops the service. |
| Latency rises near capacity because the model slows down. | Processing time is constant. Requests arrive randomly and sometimes bunch up, and near capacity there is no idle time to clear them, so the queue grows. |
| One repeated sentence is enough for a load test. | It does not represent real lengths, and caching makes the results optimistic. Long requests are what make p95. |
Try it yourself
Why does latency rise sharply near maximum capacity though each request's processing time is constant?
Requests arrive randomly, not evenly, so they sometimes bunch up. Near capacity there is no idle time to clear the bunch, so the queue grows.
A load test sends the same short sentence 10 thousand times. What is the problem?
Latency depends on input length, and long requests are what make p95. A single repeated input may be served from a cache that will not help you in production.
Quick review cards
Check Your Understanding
Your service handles 20 requests a second in theory, and the expected peak is 18. Is one server enough?
No. At 90% of capacity p95 doubles, as you saw in the simulation, and any small rise above the forecast tips it into collapse. And one server means any failure or deployment stops the whole service. Plan to run at 60% to 70% at peak, meaning at least two servers here.
In the simulation, why was p95 at 25% busy 585 ms when the average request takes 200?
Because the processing time itself is random with an exponential distribution, so some requests take several times the average even without a queue. The queue adds on top of that as you near capacity.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.