Monitoring and drift
Lesson 5 of 6
5Monitoring and drift
The big pictureA service can run without a single error, every request returning 200, while the model gets worse every week. That is drift: the incoming data gradually moves away from what the model was trained on. Monitoring starts with three things you always log and ends with a metric telling you the data changed before users tell you.
What the picture shows: After the app update, messages became far longer than what the model trained on, so the curve shifted right. The service still returns 200, and only PSI revealed that the data changed.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A model can run with no technical error at all and give worse answers every month, because the world changed: people ask new questions, prices moved. Like an old map of a city where new roads were built. Monitoring means noticing that before users complain.
You always log inputs and outputs (with privacy protected), latency, errors and the model version. drift comes in two kinds: in the input distribution (data drift) and in the relation to the target (concept drift). You measure it by comparing distributions with a reference period and alert past a threshold.
Before You Start
🌐From a notebook to a service
Your service returns the model version in every reply, the first thing you log.
⏱️Latency and cost
You know p50 and p95, and will see them on the monitoring panel.
The Core Idea
A service with no errors can get worse every week. Here you start with what to log on every request, then removing personal data, then measuring drift, then the monitoring panel and the labelled sample.
1What to log on every request
Every request writes one JSON log line. It always holds three fields: the request id, the model version and the latency.
Look at the picture: with these fields you answer two questions: what happened to this request, and did anything change this week? Add the input length and the result.
2Redact before writing
Logs are read by many teams, copied and kept for months. So personal data is removed before writing, not after, with limited retention and limited readers.
import json
import re
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
PHONE = re.compile(r"(?:\+?966|0)?5\d{8}\b")
NATIONAL_ID = re.compile(r"\b[12]\d{9}\b")
def redact(text):
text = EMAIL.sub("[email]", text)
text = NATIONAL_ID.sub("[id]", text)
return PHONE.sub("[phone]", text)
def log_request(request_id, text, label, latency_ms, version):
print(json.dumps({
"request_id": request_id,
"model_version": version,
"latency_ms": latency_ms,
"input_chars": len(text),
"input_preview": redact(text)[:60],
"label": label,
}, ensure_ascii=False))
log_request("r-001", "رقمي 0551234567 وبريدي sara@example.com والطلب لم يصل", "negative", 184, "v3")
log_request("r-002", "هويتي 1098765432 أريد تحديث العنوان", "neutral", 201, "v3")
The phone number, email and ID number were replaced before anything was written. Logs are read by many teams, copied into other systems and kept for months, so personal data in them is a leak waiting to happen. These three patterns are a start, not the end: add what is specific to your users (bank account numbers, for example), and test them on a real sample.
3Drift and PSI
Drift is a change in the input distribution from what the model trained on, so its accuracy silently drops while every request returns 200.
Look at the picture: PSI is one number you read with two cut points.
Message length at training time, then in production week one, then week eight after an app update added a long form.
import numpy as np
rng = np.random.default_rng(0)
train_len = rng.lognormal(np.log(60), 0.5, 5000) # message length at training time
week_1 = rng.lognormal(np.log(62), 0.5, 2000) # live traffic, first week
week_8 = rng.lognormal(np.log(120), 0.6, 2000) # after a new app version added a long form
def psi(expected, actual, bins=10):
edges = np.percentile(expected, np.linspace(0, 100, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected) + 1e-6
a = np.histogram(actual, edges)[0] / len(actual) + 1e-6
return float(np.sum((a - e) * np.log(a / e)))
for name, live in [("week 1", week_1), ("week 8", week_8)]:
score = psi(train_len, live)
verdict = "stable" if score < 0.1 else "watch" if score < 0.25 else "drifted: investigate"
print(f"{name}: PSI = {score:.2f} ({verdict})")
PSI (population stability index) splits values into ten bins by the training data and measures how much each bin's share changed. Week one is 0.01: stable. Week eight is 1.40: a drastic change, with the model seeing far longer messages than it trained on. The common rule: below 0.1 stable, 0.1 to 0.25 watch, above 0.25 investigate. Then take a sample and measure real accuracy before any decision.
4The panel and the labelled sample
The most dangerous drift is invisible in the inputs: the same words with a new meaning. Here you need a small periodic sample, say fifty messages a week, labelled by a person.
Look at the picture: the drift metric is an early warning, and accuracy on the sample is the truth. Do not retrain before it confirms.
from prometheus_client import Counter, Histogram
REQUESTS = Counter("predict_requests_total", "Prediction requests", ["model_version", "status"])
LATENCY = Histogram("predict_latency_seconds", "Prediction latency", ["model_version"],
buckets=[0.05, 0.1, 0.2, 0.3, 0.5, 1, 2, 5])
INPUT_CHARS = Histogram("predict_input_chars", "Input length", buckets=[20, 50, 100, 200, 500, 1000, 2000])
def observe(version, status, seconds, chars):
REQUESTS.labels(version, status).inc()
LATENCY.labels(version).observe(seconds)
INPUT_CHARS.observe(chars)
Three metrics are enough for a first panel: requests by version and status (the error rate), latency as a histogram (p50 and p95 come from it), and input length (an early drift warning). Prometheus collects them and Grafana draws them.
How It Works, Step by Step: how PSI measures drift
Split the training-time values into ten equal-count bins by percentile.
Compute each bin's share in the training data and in the current production data.
Sum, over the bins, the difference between the two shares times the log of their ratio.
Under 0.1 is stable, 0.1 to 0.25 watch, over 0.25 investigate with a labelled sample before any decision.
Worked example
Every request writes a structured (JSON) log line with at least: the request id, the model version and the latency. And with them what helps diagnosis: input length, the result, and a short preview of the input after removing personal data from it.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| No errors in the service means the model is fine. | Every request can return 200 while the model gets worse every week because the inputs moved away from what it trained on. |
| A high PSI means retrain immediately. | Drift is a warning, not a verdict. The model may still be fine, or the cause may be a collection bug. Measure accuracy on a labelled sample first. |
| Logging the full input is necessary for diagnosis. | The length and a redacted preview are usually enough. Full samples are kept with consent, in a place with specific permissions. |
Try it yourself
Which three things do you always log with every request?
The id ties a complaint to a request, the version tells you which model answered, and the latency reveals slowness. None of them is personal data.
PSI for an important feature reached 0.4. What is the next step?
Drift is a warning, not a verdict. The model may still be fine on the new distribution, or the cause may be a data collection bug rather than a real change.
Quick review cards
Check Your Understanding
PSI for input length is 0.05 (stable), but user complaints doubled. What is possible?
Drift this metric cannot see: the meaning changed (a new product, a new topic), another feature besides length changed, or the problem is not the model at all (the interface, or latency). Take a sample of the complained-about requests, label it, and measure accuracy on it. PSI on one feature sees a small part of the picture.
Why do the request and latency metrics on the panel carry the model version as a label?
Because two versions may run side by side during a rollout. The label lets you compare them directly, so you know at once if the new version raised errors or latency.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.