Observability: logs, metrics and traces
Lesson 4 of 4
4Observability: logs, metrics and traces
The big pictureOn your machine you see the error in the terminal. In production there are thousands of requests on many servers. Observability gives you three lenses: logs say what happened, metrics say how much, and traces say where the time went.
What the picture shows: The three pillars of observability. Alerts come from metrics; diagnosis comes from logs and traces.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Like a car dashboard: the speed, temperature and fuel gauges (metrics) tell you at a glance that something is off. The service log (logs) tells you what happened in detail. The workshop's computer diagnosis (traces) pinpoints exactly which part.
Structured logs (JSON) with fixed fields: time, level, message, request_id, user, duration, so they can be searched and aggregated, with no secrets or personal data. Metrics: the four golden signals (Google SRE): latency (p95), traffic (requests/s), errors (5xx rate), saturation (CPU, memory, connections). Or RED for services and USE for resources. Prometheus collects and Grafana displays. Distributed tracing: each request has a trace_id passing through services, each step a span with its duration, and the OpenTelemetry standard unifies all three. An SLO like "99.9% of requests succeed within 300ms over 30 days" gives an error budget. Alert on symptoms (the SLO is burning), not causes (CPU at 80%), and every alert must require action.
Before You Start
🚢Deployment
An app running on a server.
🏎️Performance
The p95 percentiles lesson.
The Core Idea
1Structured logs with a request id
Every line is JSON with the same fields, and every line from the same request carries the same request_id, so you search for it and see the whole story:
import json
def log(level, message, **fields):
record = {"level": level, "msg": message, **fields}
print(json.dumps(record, ensure_ascii=False))
request_id = "req-8f2a"
log("info", "order received", request_id=request_id, user_id=42, items=2)
log("warning", "payment slow", request_id=request_id, provider="pay", ms=2400)
log("error", "payment failed", request_id=request_id, status=502)2Golden signals from the logs
From the same request data you compute the four signals for one minute and decide whether to alert:
requests = [
{"ms": 120, "status": 200}, {"ms": 95, "status": 200}, {"ms": 2300, "status": 502},
{"ms": 110, "status": 200}, {"ms": 130, "status": 201}, {"ms": 2500, "status": 504},
{"ms": 105, "status": 200}, {"ms": 140, "status": 200}, {"ms": 98, "status": 200},
{"ms": 115, "status": 200},
]
cpu_percent = 64
latencies = sorted(r["ms"] for r in requests)
p95 = latencies[max(0, round(0.95 * len(latencies)) - 1)]
error_rate = sum(r["status"] >= 500 for r in requests) / len(requests)
print(f"latency p95 : {p95} ms")
print(f"traffic : {len(requests)} req/min")
print(f"errors : {error_rate:.0%}")
print(f"saturation : {cpu_percent}% cpu")
SLO_ERROR_RATE = 0.01
if error_rate > SLO_ERROR_RATE:
print("ALERT: error rate above SLO, users are affected")3Distributed tracing and OpenTelemetry
The request passes through the API, the order service, payment, then the database. A trace draws each step with its time, so you see 2 of the 2.4 seconds were spent at the payment provider. OpenTelemetry adds this automatically for most frameworks:
pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-bootstrap -a install OTEL_SERVICE_NAME=cafe-api \ OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317 \ opentelemetry-instrument uvicorn main:app
request_id in a response header. When a user sends you an error screenshot showing it, you find their logs at once.Worked example
Problem: users complain "payment hangs sometimes", CPU is 30% and there is no alert. Metrics: payment p95 at 2.4 seconds, and a 4% 5xx rate since yesterday. Trace: 85% of the time in the payment provider call. Logs (search by request_id): timeout errors from the provider after its certificate update. Fix: a 3-second timeout with one retry, and a new alert on the payment error rate, not on CPU.
Myth vs Reality
| Myth | Reality |
|---|---|
| More logs = better observability. | Logs with no structure and no id are expensive noise. Structure and context matter more than volume. |
| Alert on everything so you miss nothing. | Too many alerts make the team ignore them all. Alert on what users feel. |
Where you meet it at work
🚨On-call
A clear alert waking the right person with enough information.
📉Post-incident
Logs and traces rebuild exactly what happened.
Test your understanding
What are the four golden signals?
From the Google SRE book, covering what users feel.
Why a request_id on every log line?
Thousands of requests interleave, and the id separates them.
What do you alert on first?
Symptoms users feel matter more than possible causes.
Quick review cards
Check Your Understanding
What separates logs from metrics?
Logs are detailed per-request events; metrics are numbers aggregated over time and cheap to store.
What do you never log?
Passwords, tokens, card numbers and sensitive personal data.
What is an SLO for?
It turns "the site is slow" into an agreed number and decides when features pause in favour of stability.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.