Project: an Arabic document assistant
Lesson 10 of 10
10Project: an Arabic document assistant
The big pictureIn this project you build an assistant that answers questions from real Arabic documents, cites the source after every fact, and declines when it does not know. Most important: every change you make comes with a number before and after, in an experiment table you can defend.
What the picture shows: This is the project's first pipeline, and the dashed stop is what sets it apart: before any answer the system asks whether it found something close enough. "What does lunch cost?" stopped here and was declined, while «كم يوم اجازه سنويه؟» went through and came back with the identifier [leave.md#1].
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
You build an assistant that answers questions from real Arabic files and names, in every answer, the page it took it from, like a student citing every quote. Before any improvement you score it on fixed questions, then score it again after, so you know by the number whether it improved.
A full pipeline: load, normalise, chunk, embed and index in pgvector, then hybrid search and reranking, then generation with cited sources and refusal when none exist. With a golden set and a table logging Recall@k and faithfulness before and after every change.
Before You Start
🔤Arabic retrieval
You have one normalisation function that both documents and questions pass through.
🚧The system's limits
You know how to choose the refusal threshold from in-scope and out-of-scope data.
📊Retrieval metrics
You can compute recall and faithfulness, two columns of your experiment table.
The Core Idea
In this project you build an assistant that answers from real Arabic documents and cites its sources. You start with the documents and golden questions, then indexing, then the full pipeline, then the experiment table.
1Documents and questions first
Pick real Arabic documents whose content you know, between 20 and 100 of them. Then write the golden questions before any code.
Look at the picture: fifty questions, ten of them outside the documents, and five typed with common mistakes that expose any path that forgot normalisation.
2Indexing
Every chunk is stored twice: the original for display and citation, the normalised copy for search. And every chunk has an identifier like leave.md#1 for the answer to cite.
Look at the picture: the document is chunked by heading, then each chunk gets its two copies and its identifier.
3Answer with a source, or decline
The assistant answers from the documents only, and puts the chunk identifier after every fact so the user can check it. When the top score is below the threshold, it declines with a helpful message instead of guessing.
Five chunks, Arabic normalisation, retrieval, citing by identifier, and refusal below the threshold. Here "generation" returns the best chunk as is; in your project a language model with the template from the context engineering lesson takes its place.
import re
from sklearn.feature_extraction.text import TfidfVectorizer
docs = {
"leave.md#1": "يحق للموظف إجازةٌ سنويةٌ مدتها واحد وعشرون يوم عمل.",
"leave.md#2": "تُقدَّم طلبات الإجازة قبل أسبوعين على الأقل.",
"leave.md#3": "الإجازة المرضية تحتاج تقريراً طبياً إذا زادت عن يومين.",
"travel.md#1": "تُصرف بدلات السفر خلال ١٠ أيام من تقديم الفواتير.",
"it.md#1": "رمز الخطأ E-1043 في بوابة الحجز يعني انتهاء الجلسة.",
}
def normalize(t):
t = re.sub(r"[ً-ْٰـ]", "", t)
t = re.sub("[إأآٱ]", "ا", t).replace("ى", "ي").replace("ة", "ه")
return t.translate(str.maketrans("٠١٢٣٤٥٦٧٨٩", "0123456789"))
ids = list(docs)
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4))
matrix = vec.fit_transform([normalize(docs[i]) for i in ids])
def answer(question, threshold=0.25):
scores = (matrix @ vec.transform([normalize(question)]).T).toarray().ravel()
best = scores.argmax()
if scores[best] < threshold:
return f"لا أجد هذا في المستندات المتاحة (أعلى تشابه {scores[best]:.2f})."
return f"{docs[ids[best]]} [{ids[best]}]"
for q in ["كم يوم اجازه سنويه؟", "متى تصرف بدلات السفر؟", "ما سعر وجبة الغداء؟"]:
print(q, "\n ", answer(q))
The first question is typed without a hamza and with ha for ta marbuta, and found its document because both sides are normalised. The answer carries its identifier [leave.md#1], so the user can check it. The third question is outside the documents, so the system declined instead of offering the nearest thing. The 0.25 threshold here is for illustration; in your project you choose it the way the system limits lesson shows.
4The experiment table
Build the simplest version that works end to end and record its number. Then add one improvement per row, on the same golden set.
Look at the picture: the same retrieval under four settings on eight questions. The table answers, not intuition.
Eight golden questions, and the same retrieval under four settings: words or character n-grams, with and without normalisation.
import re
from sklearn.feature_extraction.text import TfidfVectorizer
docs = ["يحق للموظف إجازةٌ سنويةٌ مدتها واحد وعشرون يوم عمل.",
"تُقدَّم طلبات الإجازة قبل أسبوعين على الأقل.",
"الإجازة المرضية تحتاج تقريراً طبياً إذا زادت عن يومين.",
"يوافق المدير المباشر على الإجازات.",
"تُصرف بدلات السفر خلال ١٠ أيام من تقديم الفواتير.",
"يجب حجز تذاكر السفر عبر بوابة الحجز.",
"رمز الخطأ E-1043 في بوابة الحجز يعني انتهاء الجلسة.",
"العمل عن بعد مسموح يومين في الأسبوع."]
golden = [("كم يوم اجازه سنويه؟", 0), ("متى اقدم طلب الاجازه؟", 1), ("متى احتاج تقرير طبي؟", 2),
("من يوافق على اجازتي؟", 3), ("متى تصرف بدلات السفر خلال 10 ايام؟", 4),
("كيف احجز تذكرة سفر؟", 5), ("E-1043", 6), ("هل العمل من البيت مسموح؟", 7)]
def normalize(t):
t = re.sub(r"[ً-ْٰـ]", "", t)
t = re.sub("[إأآٱ]", "ا", t).replace("ى", "ي").replace("ة", "ه")
return t.translate(str.maketrans("٠١٢٣٤٥٦٧٨٩", "0123456789"))
def run(analyzer, norm):
f = normalize if norm else (lambda s: s)
vec = TfidfVectorizer(**analyzer)
m = vec.fit_transform([f(d) for d in docs])
hits = sum(int((m @ vec.transform([f(q)]).T).toarray().ravel().argmax() == t) for q, t in golden)
return hits
words = dict(analyzer="word", token_pattern=r"[^\s.،؟]+")
chars = dict(analyzer="char_wb", ngram_range=(2, 4))
print("experiment top-1")
for name, a, n in [("words, raw text", words, False), ("words, normalised", words, True),
("char n-grams, raw", chars, False), ("char n-grams, normalised", chars, True)]:
print(f"{name:30} {run(a, n)}/{len(golden)}")
Normalisation raised word search from 5 to 6, and character n-grams hit all eight with and without it, since they tolerate small spelling differences by nature. Does that mean normalisation is unnecessary with character n-grams? On these eight questions yes, and on your documents you do not know until you measure. That is the lesson: the table answers, not intuition.
How It Works, Step by Step: how to build the assistant step by step
Write 50 golden questions before any code, 10 of them outside the documents and 5 typed with common mistakes.
Build the simplest version that works end to end: keyword search, a simple template, one model, and record its number.
Add one improvement per row: normalisation, then better chunking, then vectors, then hybrid, then reranking.
Adopt the setting the table justifies, and write a sentence on why you chose it given its cost and speed.
Worked example
Pick a real set of Arabic documents you know: your university's regulations, your workplace's policies, or a product's user guides. Between twenty and a hundred documents. Then write the golden set before any code, as in the evaluation-first lesson.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Start with the strongest pipeline you can: hybrid with reranking. | Without a baseline number you cannot tell what each improvement added, and you often find some complexity adds nothing on your documents. |
| Normalisation is always needed whatever the search method. | In the small table character n-grams hit 8 of 8 with and without it, while it raised word search from 5 to 6. On your documents you do not know until you measure. |
| Changing two things at once saves time. | A change that adds three points and another that removes two look together like "one point better", and you keep the harmful change without knowing. |
Try it yourself
What is the first version you build in this project?
Without a baseline number you will not know what each improvement added. And you often discover the simple version is better than expected, and that some complexity adds nothing.
Why change one thing per row of the experiment table?
A change that adds three points and another that removes two look together like "one point better", and you keep the harmful change without knowing.
Quick review cards
Check Your Understanding
You added reranking, correctness rose from 82% to 84% and response time doubled. Do you keep it?
It depends on your users, and it is a decision you write in the table, not hide. A two-point gap may be within noise, so rerun first. If it holds, ask: will users accept a longer wait for one more correct answer in fifty? In an internal support tool usually yes; in a fast chat interface maybe not.
Why ask three people who do not know the system to ask it ten questions each?
Because their questions expose phrasings and topics your golden set did not anticipate, so you add the best of them and your number gets closer to what the system will really face.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.