Embeddings

Two ways into the same idea

Before You Start

The Core Idea

In this lesson you turn a token into a vector, measure similarity between two vectors, see where the numbers come from, search for nearest neighbours, and see where similarity misleads.

1From a number to a vector

After tokenising, every token is a number, but 512 is not closer in meaning to 513. So we give each token a row of hundreds of numbers, in a table of vocabulary size by width.

Look at the picture: the number does not go into the model as it is; it picks a row. This table is the first layer of every transformer.

2Similarity: the cosine

We measure similarity with the cosine of the angle between two vectors: the dot product divided by their two lengths. 1 means the same direction, 0 means unrelated.

Let us try it on four words, each with only three dimensions whose meaning we know.

import numpy as np

# three made-up dimensions: [royalty, person, food]
vec = {
    "king":  np.array([0.9, 0.8, 0.0]),
    "queen": np.array([0.9, 0.7, 0.1]),
    "man":   np.array([0.1, 0.9, 0.0]),
    "bread": np.array([0.0, 0.0, 0.9]),
}

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

for w in ["queen", "man", "bread"]:
    print(f"king vs {w:5}: {cosine(vec['king'], vec[w]):.3f}")

"king" and "queen" are almost identical, "man" is close because it shares the "person" dimension, and "bread" is zero because it shares none. In real embeddings there are hundreds of dimensions and nobody names them, but the computation is exactly the same.

3A word is known by its company

In the example above we chose the dimensions and their meaning ourselves. Look at the picture: in real training the points start scattered, then words used alike gather together.

Nobody wrote that "king" = [0.9, 0.8, 0.0]. The model starts with random numbers and adjusts them during training. The idea that makes this work is old: "you shall know a word by the company it keeps". Words that appear in similar contexts end up with similar vectors.

4Nearest neighbours

Nearest neighbours are the vectors most similar to a given one. All of semantic search is this: give me the nearest neighbours of my question's vector.

We count how often each word appears near each other one (a two-word window), compress the table to four dimensions with SVD, and look up each word's nearest neighbour. No neural network, and meaning still appears.

import numpy as np

corpus = """
the cat drinks milk . the dog drinks water . the cat eats fish .
the dog eats meat . a kitten drinks milk . a puppy drinks water .
the car needs fuel . the truck needs fuel . a car has wheels .
a truck has wheels . the kitten eats fish . the puppy eats meat .
""".split()
vocab = sorted(set(corpus) - {".", "the", "a"})
idx = {w: i for i, w in enumerate(vocab)}

counts = np.zeros((len(vocab), len(vocab)))
for i, w in enumerate(corpus):
    for j in range(max(0, i - 2), min(len(corpus), i + 3)):
        if j != i and w in idx and corpus[j] in idx:
            counts[idx[w], idx[corpus[j]]] += 1

u, s, _ = np.linalg.svd(np.log1p(counts))
emb = u[:, :4] * s[:4]
emb /= np.linalg.norm(emb, axis=1, keepdims=True)

nouns = ["cat", "dog", "kitten", "puppy", "car", "truck", "milk", "water", "fish", "meat"]

def nearest(word, k=2):
    sims = {w: float(emb[idx[w]] @ emb[idx[word]]) for w in nouns if w != word}
    return sorted(sims, key=sims.get, reverse=True)[:k]

for w in ["cat", "puppy", "car", "milk"]:
    print(w, "->", nearest(w))

"cat" is closest to "kitten", "puppy" to "cat" and "dog", and "car" to "truck", though nobody told the code any meaning. The second neighbour is sometimes noise (water next to car) because the text is only 48 words. Real embedding models do something cleverer over billions of words, but this is the principle.

5Where similarity misleads

An embedding measures similarity of use, not of meaning. So opposites sit very close, because they appear in the same sentences.

In a small text, "hot" and "cold" appear in exactly the same places.

import numpy as np

corpus = ("the tea is hot today . the tea is cold today . "
          "the soup is hot now . the soup is cold now . "
          "the room is warm today . the room is big now . ") * 3
words = corpus.split()
vocab = sorted(set(words))
idx = {w: i for i, w in enumerate(vocab)}
counts = np.zeros((len(vocab), len(vocab)))
for i, w in enumerate(words):
    for j in range(max(0, i - 2), min(len(words), i + 3)):
        if j != i:
            counts[idx[w], idx[words[j]]] += 1
emb = counts / np.linalg.norm(counts, axis=1, keepdims=True)

def sim(a, b):
    return float(emb[idx[a]] @ emb[idx[b]])

print(f"hot vs cold: {sim('hot', 'cold'):.2f}   (opposites)")
print(f"hot vs warm: {sim('hot', 'warm'):.2f}   (close in meaning)")
print(f"hot vs tea:  {sim('hot', 'tea'):.2f}")

"hot" and "cold" are identical (1.00) because their company is the same, and closer than "hot" and "warm". Large models soften this but do not remove it. In a search system, "which drugs are banned?" can confidently retrieve the paragraph on "allowed drugs". That is why we add keyword search and reranking in the hybrid search lesson.

Look at the picture: the two opposites are identical, and closer than two words with nearly the same meaning.

How It Works, Step by Step: how embedding vectors come about

Going deeper

An embedding model that is good for English is not necessarily good for Arabic. Before you build a search system, take twenty questions from your domain with the right paragraph for each, and measure how often the right paragraph is in the top three. This small test picks your model better than any general leaderboard.

Worked example

Imagine each word has only three numbers: how royal it is, how much a person, how much a food. Similar words will have close numbers. We measure similarity between two vectors by the cosine of the angle between them: 1 means the same direction, 0 means unrelated.

Example: Sentence embeddings with a real model

This is what you will actually write. It needs sentence-transformers and a multilingual model download, so it is here for reading.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-small")
docs = ["passage: مواعيد الدوام من الثامنة إلى الرابعة",
        "passage: يمكن استرجاع المبلغ خلال أربعة عشر يومًا"]
query = "query: متى أستطيع إرجاع المنتج؟"

d = model.encode(docs, normalize_embeddings=True)
q = model.encode(query, normalize_embeddings=True)
print(d @ q)    # the second passage should score higher

normalize_embeddings=True makes every vector length 1, so the dot product is the cosine itself. Some models such as E5 expect the query: and passage: prefixes, and forgetting them silently lowers quality. Always read the model card.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding