Training pipelines and experiment tracking
Lesson 1 of 3
1Training pipelines and experiment tracking
The big picture"The good model was in Tuesday's run, but I do not remember the settings." The most famous sentence in machine learning teams. MLOps starts with recording every run automatically.
What the picture shows: A reproducible pipeline: every input has a version, and every output is recorded with what made it.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Like a chef testing recipes: without writing down the amounts, temperature and time for each attempt, they will never repeat the best dish. Experiment tracking is the chef's notebook, but written automatically.
Reproducibility: the code commit, the data version (DVC, a file hash or a time-versioned table), the environment (pinned requirements or a Docker image), and the seeds (random, numpy, torch), noting some GPU operations are non-deterministic. Experiment tracking (MLflow, Weights & Biases): parameters, metrics across epochs, artefacts (the model, the confusion matrix) and tags. A pipeline is separate steps: prepare, split, train, evaluate, register, driven by one config file, and run in CI or on a schedule. A model registry: versions with stages (staging, production) and a model card documenting the data, metrics and limits.
Before You Start
🧠Classical ML
You have trained and evaluated a model.
🚀Serving
The model serving track.
The Core Idea
1An experiment log yourself
Before the tools, the idea: every run records the data fingerprint, the settings and the result. Here is a simple, deterministic "training" and an in-memory log:
import hashlib
import json
import random
DATA = [(x, 2 * x + 1) for x in range(20)]
runs = []
def fingerprint(data):
return hashlib.sha256(json.dumps(data).encode()).hexdigest()[:10]
def train(lr, epochs, seed):
rng = random.Random(seed)
w, b = rng.random(), rng.random()
for _ in range(epochs):
for x, y in DATA:
err = (w * x + b) - y
w -= lr * err * x
b -= lr * err
loss = sum(((w * x + b) - y) ** 2 for x, y in DATA) / len(DATA)
return round(loss, 6)
for lr in (0.001, 0.003):
params = {"lr": lr, "epochs": 30, "seed": 42}
runs.append({"data": fingerprint(DATA), **params, "loss": train(**params)})
best = min(runs, key=lambda r: r["loss"])
for r in runs:
print(r)
print("best:", best["lr"], "reproduced:", train(best["lr"], 30, 42) == best["loss"])2MLflow
The same idea with a real tool: each start_run logs parameters, metrics and the model, and you compare runs in a web UI:
import mlflow
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
mlflow.set_experiment("reviews-sentiment")
with mlflow.start_run():
params = {"C": 0.5, "max_iter": 500}
mlflow.log_params(params)
mlflow.set_tag("data_version", "reviews-2026-10")
model = LogisticRegression(**params).fit(X_train, y_train)
mlflow.log_metric("accuracy", accuracy_score(y_test, model.predict(X_test)))
mlflow.sklearn.log_model(model, "model", registered_model_name="sentiment")
# mlflow ui -> http://localhost:50003Versioning data
Data changes, and the model changes with it. Record the data fingerprint with every run, and for big files use DVC, which keeps a small pointer in Git and the large file in separate storage:
pip install dvc dvc init dvc add data/reviews.csv # creates data/reviews.csv.dvc (tracked by git) git add data/reviews.csv.dvc .gitignore git commit -m "Track reviews dataset v1" dvc remote add -d store s3://my-bucket/dvc dvc push
Worked example
Problem: the model is 79% accurate in production and the team remembers 87%. Investigation: the MLflow log shows 87% was on reviews-2026-08 data with seed 1, and the deployed model trained on reviews-2026-10 after a cleaning change stripped emoji. Fix: retrain with the recorded pipeline, and add a pipeline check that rejects any model less accurate than the current version.
Myth vs Reality
| Myth | Reality |
|---|---|
| MLOps is only for big companies. | One person needs to know which settings made the best model. A simple log is enough to start. |
| Fixing the seed makes everything deterministic. | Some GPU and parallel operations are non-deterministic. Record everything and expect small differences. |
Test your understanding
What do you record with each run?
Without them you cannot rebuild the model.
Why not put a 10GB data file in Git?
DVC keeps a small pointer in Git and the file in separate storage.
What is a model registry?
It tells you which version is in production and how it was made.
Quick review cards
Check Your Understanding
Why a script instead of a notebook for final training?
A script runs from scratch in a fixed order; a notebook may carry hidden state.
What does a data fingerprint (hash) do?
It reveals any change in the data, even one character, and ties it to the run.
What check do you add to the pipeline before deploying?
Rejecting the new model if its metrics are worse than the current one on the same test data.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.