Shipping and limits
Lesson 3 of 3
3Shipping and limits
The big pictureA system is not ready when it works for you, but when three people who do not know you use it without you explaining. This lesson is about the last step: the stranger test, a limits document that says plainly what your system cannot do and when it fails, protecting the data of whoever uses it, and a final demo that shows the numbers and the failures together.
What the picture shows: The "two questions at once" task failed for two of three, the same slice that showed up in error analysis. When automatic evaluation and real users agree on one weakness, it becomes the most important line in the limits document.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A system is not finished until people who do not know you, and will not flatter you, use it. You give it to three of them and watch where they stumble. Then you write an honest page: what your system does well, what it does not, and when nobody should trust it.
Testing with real users, logging tasks and errors, then a model card stating intended use, data, metrics, limits, risks and known biases. With a working link and reproduction instructions.
Before You Start
🏗️Building and evaluating
You have a working version, a test number measured once, and you know the slices where it fails.
📡Monitoring and drift
You have seen how personal data is redacted before a log line is written.
The Core Idea
The last step has four parts, in order: the stranger test, then the limits document, then protecting data in the logs, then a final demo that shows the numbers and the failures together.
1The stranger test
A system is ready when three people who do not know you use it without you explaining. You avoid its failures without noticing; a stranger finds them fast.
Look at the picture: three strangers as rows, five tasks as columns. You watch each cell in silence and record: did they finish, how long did it take, and where did they stumble.
Fifteen tasks are not enough for a precise number: the range runs from 48% to 89%. Their purpose is to find what fails, and the "two questions at once" task failed for two of three.
2The limits document
The limits document is part of the product, not an admission of weakness. Whoever uses your system needs to know when to trust it and when to check for themselves.
Look at the picture: accuracy by slice, which is what the document states instead of a general promise that the system is "smart".
One page that ships with the system, read by the user before relying on it.
# Limits: shipping reply drafts for <shop name> (v1.2, 2026-10)
Built for: short Arabic support messages about order status
Not built for: refunds, complaints, legal questions, other languages
beyond basic English
How well it works (150 frozen real messages, labelled by the owner):
shipping precision 96% (95% range 91% .. 98%) bar: 95%
drafts sent unedited 64% (owner's log, 2 weeks) bar: 60%
keyword baseline 85% accuracy, same set
Where it fails:
- messages with two requests: 45% accuracy -> always routed to a human
- Gulf dialect spelling not seen in training: ~82%
- names and street addresses are NOT removed from logs
When unsure (confidence < 0.7) it drafts nothing and flags the message.
Data: messages kept 30 days, phones/emails/IBANs/IDs removed before
logging, never used for training without the owner's consent.
Contact for errors: <owner email>
Every number with its range, every known weakness written down with what the system does about it, and what is not removed from the logs stated plainly. This page is what anyone assessing your project will read before the code, and it is what shows you understand your system.
The same idea is behind the Model Cards that ship with published models: intended use, performance by slice, and known limitations.
3Privacy starts in the logs
Most leaks in small projects come from logs that store every message as is. Look at the picture: every message passes through a redaction function before it is written, not after.
A small function every message passes through before it reaches the log: email, IBAN, Saudi mobile number, and national ID number.
import re
PATTERNS = [
("EMAIL", re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")),
("IBAN", re.compile(r"\bSA\d{22}\b")),
("PHONE", re.compile(r"(?<!\d)(?:\+?966|0)?5\d{8}(?!\d)")),
("NATIONAL_ID", re.compile(r"\b[12]\d{9}\b")),
]
def redact(text):
found = []
for name, pattern in PATTERNS:
text, n = pattern.subn(f"[{name}]", text)
found += [name] * n
return text, found
logs = [
"طلبي ما وصل، جوالي 0551234567 وإيميلي sara.m@example.com",
"حوّلوا المبلغ على SA0380000000608010167519",
"رقم هويتي 1012345678 وأبي أغيّر العنوان",
"كم سعر التوصيل لأبها؟",
]
for line in logs:
clean, found = redact(line)
print(clean, " <-", ", ".join(found) or "nothing")
The text stays useful for analysis, since you still know the message is about a late order, but nobody reading the log can contact its sender. The order matters: IBAN before phone, so the phone pattern does not catch part of an account number. And regular expressions do not catch everything; names and addresses need stronger tools, which is why this limit also goes in the limits document.
If the data is about people in the EU, or in Saudi Arabia under the Personal Data Protection Law, there are legal duties around consent and deletion on request.
4The final demo
The demo tells the story in a fixed order. Look at the picture: the problem and the user, then the system on a real case, then the numbers against the bar and the baseline, then a case where it fails.
Show the failure before someone discovers it. A message with two requests goes to a human because the system gets those wrong, and that builds trust no perfect demo can.
How It Works, Step by Step: how to run the stranger test
Write five tasks in advance, and find three people who know neither you nor the system.
Watch in silence, and record for each task whether it was completed, how long it took, and where the user stumbled.
If you have to step in, record the task as failed with what needed explaining.
Group the failures by task, and move what repeats into the limits document with what the system does about it.
Going deeper
Privacy starts in the logs, not in the privacy policy. Most data leaks in small projects come from logs that store every message as is, phone numbers, ID numbers and bank accounts, in a file anyone with server access can read. Remove this information before it is written, not after. And if it is data about people in the EU, or in Saudi Arabia under the Personal Data Protection Law, there are legal obligations around consent and deletion on request.
Worked example
You know how your system is "supposed" to be used, so you avoid its failures without noticing. A stranger does not know, so they find the failures fast. Three users, five tasks each, and you watch silently and write down: did they finish the task, and how long did it take.
1. Stranger test results
import math
def wilson(k, n, z=1.96):
p = k / n
c = (p + z * z / (2 * n)) / (1 + z * z / n)
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return c - h, c + h
# three strangers, five tasks each, observed without help
sessions = {
"user A": [("track order", True, 40), ("ask about size", True, 25), ("return item", False, 180),
("report damage", True, 60), ("two questions at once", False, 150)],
"user B": [("track order", True, 30), ("ask about size", True, 20), ("return item", True, 90),
("report damage", True, 45), ("two questions at once", False, 200)],
"user C": [("track order", True, 35), ("ask about size", False, 120), ("return item", True, 70),
("report damage", True, 50), ("two questions at once", True, 110)],
}
tasks = [t for s in sessions.values() for t in s]
done = sum(ok for _, ok, _ in tasks)
low, high = wilson(done, len(tasks))
print(f"tasks completed: {done}/{len(tasks)} = {done / len(tasks):.0%} (95% range {low:.0%} .. {high:.0%})")
failed_on = {}
for user, s in sessions.items():
for name, ok, _ in s:
if not ok:
failed_on.setdefault(name, []).append(user)
print("failures by task:")
for name, users in sorted(failed_on.items(), key=lambda kv: -len(kv[1])):
print(f" {name:<22} {len(users)}/3 ({', '.join(users)})")
Eleven of fifteen tasks, but the range runs from 48% to 89%, so fifteen tasks are not enough for a precise number, and that is not their purpose. Their purpose is to find what fails: the "two questions at once" task failed for two of three, which is the same slice that showed up in error analysis. When automatic evaluation and real users agree on the same weakness, that is the most important line in the limits document.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A limits document admits weakness and weakens trust. | It is part of the product: it tells the user when to trust it and when to check for themselves, making the system safer and more usable. |
| Privacy starts with writing a privacy policy. | It starts in the logs. Most leaks in small projects come from logs storing every message as is. What is never written cannot leak. |
| A good demo shows the system without mistakes. | Show the failure case yourself before someone finds it, with what the system does about it. That builds trust no perfect demo can. |
Try it yourself
Where should the customer's phone number be removed from the message?
What is never written cannot leak. Deleting later leaves a time window, backups and logs copied elsewhere, all carrying the number.
What should the limits document say about a slice where the system fails?
A general sentence does not help the user decide when to check for themselves. The specific slice, its number and the system's behaviour on it are what make the document usable.
Quick review cards
Check Your Understanding
A user in the stranger test wrote a message in a way you did not expect and the system failed. You explained how to write it and it succeeded. How do you record this task?
Failed, with a note on what needed explaining. The real user will not have anyone beside them to explain. And that note is worth more than the success, because it tells you either about a slice that needs handling, a limit to write in the limits document, or an interface that needs to make expectations clearer.
In the redaction function, why does the IBAN pattern come before the phone pattern?
So the phone pattern does not catch part of the account number. Once the whole IBAN is replaced, none of its digits are left for another pattern to match.
Fifteen tasks gave a range of 48% to 89%. Why are they still useful?
Because their purpose is not a precise number but finding what fails. And they showed the two-questions task failed for two of three.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.