Audio
Lesson 6 of 7
6Audio
The big pictureSound is a wave: thousands of numbers per second. Models usually do not read it raw; they turn it into a picture called a spectrogram, then apply the same tools as vision. For Arabic the biggest challenge is not technique but data: many dialects, few recordings.
What the picture shows: The mel curve climbs fast at low frequencies and then flattens, while plain Hz is a straight line. That is how a model spends more resolution where speech lives, just as the ear tells 100 from 200 Hz but barely 7000 from 7100.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Sound is a wave vibrating in air. For a model to understand it, we turn it into a picture like a heat map: across is time, up is pitch from low to high, and colour is loudness. Then the model can "see" speech. Arabic dialects are still a challenge for many models.
The wave is sampled at a rate like 16 kHz, then a short-time Fourier transform STFT gives the spectrum, and the mel scale mimics the ear. Speech recognition models like Whisper turn the spectrogram into text, measured by word error rate WER.
Before You Start
🖼️An image as a tensor
You know an image is a grid of numbers, and a spectrogram is a picture of sound that the same tools work on.
🧮Arrays with NumPy
You can cut an array into windows and apply an operation to each row, like np.fft.rfft here.
The Core Idea
Sound is a long wave of numbers. Here you start with the wave and the sample rate, then frequencies, then the spectrogram that turns sound into a picture, then the mel scale, then measuring speech recognition.
1The wave and the sample rate
A microphone measures air pressure thousands of times a second, giving numbers between -1 and 1. The number of measurements per second is the sample rate.
Look at the picture: speech is 16 thousand per second, music 44.1 thousand, and phone calls 8 thousand.
The rate must match what the model was trained on, or it hears the audio sped up or slowed down. And one second of speech is 16,000 numbers, far too long for a model to read raw.
2Frequencies: the Fourier transform
To know what is in a sound we ask: which frequencies does it contain, and how strongly? The fast Fourier transform, FFT, answers that question.
import numpy as np
rate = 16000
t = np.arange(rate) / rate
wave = 0.6 * np.sin(2 * np.pi * 440 * t) + 0.3 * np.sin(2 * np.pi * 1200 * t)
print("samples in one second:", len(wave))
spectrum = np.abs(np.fft.rfft(wave))
freqs = np.fft.rfftfreq(len(wave), d=1 / rate)
top = np.argsort(spectrum)[-2:]
print("strongest frequencies (Hz):", sorted(freqs[top].astype(int).tolist()))
We built a sound from two tones, 440 Hz (the note A) and 1200 Hz, then analysed it with the fast Fourier transform, FFT. It found both frequencies exactly. Speech is far richer than two tones, but the idea is the same: every sound is a sum of frequencies at different strengths.
3The spectrogram: a picture of sound
Speech changes every moment, so one spectrum for the whole file is not enough. We cut the audio into short consecutive windows and analyse each one alone.
Look at the picture: each 25 ms window advances 10 ms and becomes a column of frequencies. The result is a grid: time on one axis, frequency on the other.
Half a second at 300 Hz, then half a second at 2000 Hz. We cut the audio into windows of 400 samples (25 ms) with a hop of 160 (10 ms) and analyse each window.
import numpy as np
rate = 16000
t = np.arange(rate) / rate
wave = np.where(t < 0.5, np.sin(2 * np.pi * 300 * t), np.sin(2 * np.pi * 2000 * t))
frame, hop = 400, 160
frames = np.stack([wave[i:i + frame] * np.hanning(frame) for i in range(0, len(wave) - frame, hop)])
spec = np.abs(np.fft.rfft(frames, axis=1))
freqs = np.fft.rfftfreq(frame, d=1 / rate)
print("spectrogram shape (time, freq):", spec.shape)
for i in [10, 40, 60, 90]:
print(f"frame {i:2} at {i * hop / rate:.2f}s -> peak {freqs[spec[i].argmax()]:.0f} Hz")
98 time windows and 201 frequencies per window: a picture. Before 0.5 seconds the peak is at 320 Hz, after it at 2000. Why 320 and not 300? A 400-sample window resolves frequencies only to 16000 / 400 = 40 Hz, so the nearest bin to 300 is 320. Longer windows give finer frequency and coarser time, an unavoidable trade-off.
4The mel scale
The ear tells differences at low frequencies far better than at high ones. The mel scale compresses high frequencies to resemble human hearing.
Speech models usually use 80 mel bands, far fewer than 201 raw frequencies.
import numpy as np
def hz_to_mel(f):
return 2595 * np.log10(1 + f / 700)
for low, high in [(100, 200), (1000, 1100), (7000, 7100)]:
print(f"{low:5}-{high:5} Hz: {hz_to_mel(high) - hz_to_mel(low):6.1f} mel apart")
The same 100 Hz gap is 133 mels at low frequencies and only 14.5 at 7000. The ear clearly hears 100 versus 200 Hz and barely tells 7000 from 7100. Speech models mimic that, spending more resolution where speech lives.
5Speech recognition and error rate
You rarely build a speech model from scratch. Whisper and its peers trained on hundreds of thousands of hours, and your real job is measurement.
The measure is word error rate, WER: insertions, deletions and substitutions divided by the number of words in the reference.
This code needs the transformers library and a model download, so it is here for reading.
from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
result = asr("voice_note.wav", generate_kwargs={"language": "arabic"})
print(result["text"])
Three lines. Set the language explicitly, since left to guess the model may write an Arabic sentence in Latin letters or translate it. Then record twenty real clips from your users, transcribe them correctly by hand, and compute word error rate with the jiwer library. That number decides whether the model is fit, not the demo.
How It Works, Step by Step: how sound becomes a picture a model reads
Record the wave at 16,000 samples a second, the rate most speech models expect.
Cut it into windows of 400 samples (25 ms) that advance 160 samples (10 ms) each time.
Apply the Fourier transform to each window, so it becomes a column of frequencies and one second becomes a 98 × 201 picture.
Compress the frequencies into 80 mel bands, which shrinks the picture and brings it closer to human hearing.
Worked example
A microphone records air pressure 16,000 times a second (16 kHz, the standard for speech recognition). One second of audio is 16,000 numbers. To know what is in the sound, we ask: which frequencies does it contain?
librosa.resample or torchaudio.functional.resample first.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| The sample rate does not matter as long as the file opens. | An 8 or 44.1 kHz recording fed to a 16 kHz model is heard distorted, and the transcript is bad with no error message. Resample first. |
| A spectrogram removes noise. | The noise stays in it. Its benefit is that it is far shorter than the wave and shows the frequencies that distinguish sounds directly. |
| The error rate on news clips is the model's performance. | News clips are clear MSA on a good microphone. The real number is the word error rate on recordings like the ones your users send, in their dialects and their noise. |
Try it yourself
A 512-sample window on 16,000 Hz audio. Roughly what frequency resolution?
16000 / 512 ≈ 31 Hz between frequency bins. A longer window resolves frequency better but covers more time (32 ms here) and blurs fast events.
Why do speech models usually turn audio into a mel spectrogram instead of using the raw waveform?
One second is 16,000 numbers as a wave and 100 columns by 80 bands as a mel spectrogram. Speech sounds are distinguished by their frequencies, so the spectrogram hands that information over instead of making the model discover the Fourier transform itself. Noise stays in the spectrogram; it is not removed automatically.
Quick review cards
Check Your Understanding
A speech model scores 8% WER on news clips and 35% on users' voice notes. Likely explanations, and what do you do?
News clips are clear MSA on a good microphone; voice notes are dialects, noise, fast speech and mixed-in English words. Do: break down the errors to see which dominate (dialect? noise? names?), try a larger model or one tuned for your dialect, add common names as a prompt hint, and know that fine-tuning on a few hours of your recordings can cut errors a lot. Most importantly: 35% is your real number, not 8%.
In the hand-made spectrogram the 300 Hz peak showed up at 320. Why?
Because a 400-sample window at 16,000 Hz separates frequencies only every 40 Hz, and the nearest bin to 300 is 320. A longer window resolves finer but covers more time.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.