Project: messy Arabic data
Lesson 9 of 9
9Project: messy Arabic data
The big pictureThe first track's project: a real Arabic dataset that comes out clean, with a report defending every row you dropped and every cell you filled.
What the picture shows: Seven spellings of city names go into the normalise function, and under each one sits what it becomes. Diacritics, tatweel, the trailing space, hamza forms and ta marbuta all disappear, leaving three values you can safely group on.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Real data arrives dirty: names spelt different ways, dates in many formats, empty cells and duplicates. Cleaning is like washing vegetables before cooking. In this project you do not just clean; you write why you removed each row, so whoever reads your results can trust you.
Cleaning steps: normalise Arabic text (hamzas, taa marbuta, diacritics and digits), convert types and dates, remove duplicates, and handle missing values and outliers. You log every step with row counts before and after in a re-runnable report, so the same result comes out on a fresh copy of the data.
Before You Start
🗂️Frames and missing values
You use groupby and merges and know the difference between dropping and filling missing values, from the frames lesson.
📊The three plots
You can draw a histogram, a scatter and a box plot, and apply the interquartile rule from the plots lesson.
The Core Idea
The project goes through four steps in order: normalise the Arabic text, then handle duplicates and missing values, then three plots, then a report that defends every decision.
1Normalise the Arabic text first
Pick a real Arabic dataset with text, numbers and missing values, small enough to inspect in one sitting. Small and understood beats large and unknown.
The first thing you do with it is normalise: unify hamza forms, alef maksura and ta marbuta, strip diacritics and tatweel, and collapse whitespace.
import re
import unicodedata
HARAKAT = re.compile(r"[\u064B-\u0652\u0640]")
def normalise(text):
text = unicodedata.normalize("NFKC", text.strip())
text = HARAKAT.sub("", text)
text = text.replace("أ", "ا").replace("إ", "ا").replace("آ", "ا")
text = text.replace("ى", "ي").replace("ة", "ه")
return re.sub(r"\s+", " ", text)
raw = ["الرِّياض", "الريَاض ", "الـرياض", "جدة", "جده", "أبها", "ابها"]
cleaned = [normalise(value) for value in raw]
print(cleaned)
print(len(set(raw)), "->", len(set(cleaned)))
Seven spellings became three values. Note that one form carried tatweel, another carried diacritics, and a third a trailing space; all three were separate rows in any grouping before this function. Note too that unifying ta marbuta with ha collapsed two spellings of Jeddah into one value.
2Then duplicates, then missing values
The order is not optional. Look at the picture: normalise before dropping duplicates, because "جدة" and "جده" are different strings and the same city.
import pandas as pd
frame = pd.DataFrame({
"city": ["الرياض", "الرياض", "جده", "جدة", "الدمام"],
"orders": [120, 120, 90, 90, 40],
"revenue": [4800.0, 4800.0, 3150.0, 3150.0, None],
})
print("rows in:", len(frame))
frame["city"] = frame["city"].str.replace("ة", "ه", regex=False)
before = len(frame)
frame = frame.drop_duplicates()
print("dropped duplicates:", before - len(frame))
missing = int(frame["revenue"].isna().sum())
print("missing revenue:", missing)
per_order = (frame["revenue"] / frame["orders"]).dropna()
median_rate = float(per_order.median())
frame["revenue"] = frame["revenue"].fillna(frame["orders"] * median_rate)
print("median per order:", round(median_rate, 2))
print("rows out:", len(frame), "missing after:", int(frame["revenue"].isna().sum()))
print("total revenue:", round(float(frame["revenue"].sum()), 2))
Five rows in, three rows out. Duplicates were dropped after normalising, not before, and the missing revenue was filled with the median revenue per order times the order count, a value derived from similar rows rather than a zero. That decision belongs in your report as one line: "filled Dammam's revenue with the median revenue per order of 37.5 because its order count is known".
3Three plots, one line each
Draw the three plots from the earlier lesson, each with its own question. Look at the picture: the histogram for spread, the scatter for two columns, the box plot for outliers.
A plot without its line of interpretation does not count. The line ties what you saw to a decision you made in the data.
4The report: what, how many rows, why
The report is a three column table: what I did, to how many rows, and why. Look at the picture: the numbers this project's report states.
Every drop is a hypothesis about your data. In a month you will not recall why three hundred rows vanished, and your reader will not trust a number with no provenance.
One page is enough. Honesty matters more than length.
How It Works, Step by Step: how to clean the dataset in the right order
Normalise the text first: unify hamza forms, alef maksura and ta marbuta, and strip diacritics, tatweel and extra spaces.
Drop duplicates after normalising, so "جدة" and "جده" are caught as one duplicated row.
Handle missing values with a value derived from similar rows, such as the median revenue per order times the order count.
Write every decision into the report: what you did, to how many rows, and why.
Worked example
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| The order of cleaning steps does not matter as long as they all run. | Dropping duplicates before normalising leaves "جدة" and "جده" as two different rows, so the duplicate survives and breaks every grouping after it. |
| A row with one empty cell should be dropped. | In a ten column table that can remove half the data, usually not at random, so you lose a whole category because of one column. |
| The report is an optional extra after cleaning. | The report is part of the delivery: in a month you will not recall why three hundred rows vanished, and your reader will not trust a number with no provenance. |
Try it yourself
You grouped orders by city and three spellings of the same city appeared as three cities. When should you have intervened?
Normalisation comes first. Merging by hand after grouping fixes this case and leaves the next one, and becomes manual work repeated on every data refresh. Encoding is a different problem entirely: a wrongly encoded file gives you mangled characters, not several spellings of one name.
Quick review cards
Check Your Understanding
why fill with a median rather than a mean, usually?
Because the mean is dragged by outliers, as the plotting lesson showed: one huge value raises it and makes every fill you derive from it too large. The median stays in the actual middle of the distribution. Better than either, where possible, is a value computed from similar rows, as we did here with revenue per order.
In the second example five rows went in and three came out. Where did the difference come from?
From two duplicate rows dropped after normalising: the repeated Riyadh row, and the Jeddah row written once as "جده" and once as "جدة". The Dammam row with missing revenue was filled, not dropped.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.