Word Embeddings

Every word becomes a point in a high-dimensional space where similar meanings sit close together and directions encode relationships. Explore neighbours, analogies and cosine similarity.

Intermediate

These 45 vectors are hand-crafted and illustrative (16 dimensions with readable meanings). Real embeddings are learned from text and have hundreds of unlabelled dimensions, but they show the same kinds of structure.

PC1 (21% of variance)PC3 (16%)kingqueenprinceprincessmanwomanboygirlfathermothercatkittendogpuppyliontigerwolffoxhorsecowelephantfranceparisgermanyberlinitalyromespainmadridjapantokyochinabeijingwalkwalkedrunranswimswamgowenteatateseesaw
People and royaltyAnimalsCountries and capitalsVerbs and tensesanalogy offset (solid: b → a, dashed: same offset from c)

The 16-D vectors are squashed to 2-D, so distances on the plot are only approximate. Cosine similarities in the panels are computed on the full vectors. Try fitting PCA to one group to see its internal structure.

Nearest neighbours

Click a word on the plot or pick one here.

  1. 0.875
  2. 0.833
  3. 0.815
  4. 0.718
  5. 0.625
  6. 0.625

Analogy: a − b + c = ?

king − man + woman ≈ queen

  1. queen
    0.983
  2. princess
    0.855
  3. mother
    0.822
  4. girl
    0.632
  5. prince
    0.605

The offset b → a (for example man → king, "add royalty") is added to c. The result is a point in space that is not exactly any word, so we report the words with the highest cosine similarity, excluding a, b and c.

Cosine similarity, step by step

Dimensionxyx·y
person0.050.07
male (+) / female (-)0.030.08
royalty-0.00-0.02
young (+) / older (-)0.041.06
animal1.511.43
domestic (+) / wild (-)1.070.93
feline (+) / canine (-)0.920.94
size-0.56-0.95
place-0.020.04
country (+) / capital (-)0.030.03
west (-) / east (+)0.020.08
south (-) / north (+)-0.010.07
verb0.01-0.07
past tense0.06-0.04
verb sense A-0.010.04
verb sense B-0.030.03

x · y = Σ xᵢyᵢ = 4.599

|x| = 2.151, |y| = 2.414

cos θ = 4.599 / (2.151 × 2.414) = 0.886

θ ≈ 27.7°

How to use this visualizer

  1. 1
    Look at the projection: the four colours (people, animals, places, verbs) form separate clusters.
  2. 2
    Click any word to list its nearest neighbours by cosine similarity. Lines connect it to its three closest words.
  3. 3
    Pick an analogy preset such as paris − france + italy. The plot re-fits to that group and draws the offset from b to a, then the same offset starting at c.
  4. 4
    Choose your own a, b and c in the analogy tool and see which word lands nearest to the result.
  5. 5
    Use the cosine panel to compare two words dimension by dimension: green bars add to the similarity, red bars subtract from it.
  6. 6
    Switch the PCA fit and axes to see how much a 2-D view hides.

What are word embeddings?

A language model cannot do arithmetic on the string "cat". Instead, each token in the vocabulary gets a row in an embedding matrix: a list of numbers (a vector) that the model learns during training. Models like word2vec and GloVe learn these vectors by predicting which words appear near each other, and modern transformers learn them as their very first layer.

Training pushes words that appear in similar contexts towards similar vectors. A side effect is that relationships become directions: the step from "man" to "king" is roughly the same as the step from "woman" to "queen". That is why vector arithmetic can solve analogies.

Distributional hypothesis

Words used in similar contexts get similar vectors. "cat" and "kitten" appear near the same words, so they end up close.

Directions carry meaning

Consistent offsets such as male to female or present to past tense let simple vector arithmetic answer analogies.

Cosine, not distance

Similarity is usually measured by the angle between vectors, so a frequent word with a long vector is not penalised.

Projection loses detail

PCA keeps the directions of largest variance. Words that look close in 2-D can be far apart in the full space.

Key formulas

Cosine similarity

cos θ = (x · y) / (|x| |y|)

x · y
dot product, Σ xᵢyᵢ
|x|
length of x, √(Σ xᵢ²)
θ
angle between the vectors; cos θ ranges from −1 (opposite) to 1 (same direction)

Analogy by vector arithmetic

answer = argmaxw ∉ {a, b, c} cos(vw, va − vb + vc)

PCA projection

(p₁, p₂) = ((v − μ) · u₁, (v − μ) · u₂)

μ
mean vector of the fitted words
u₁, u₂
top eigenvectors of the covariance matrix C = (1/n) Σ (v − μ)(v − μ)ᵀ

Why the arrows form a parallelogram

PCA is a linear map, so projecting a − b + c gives exactly proj(a) − proj(b) + proj(c). The two arrows on the plot are the same offset drawn twice, and the open circle marks where the target vector lands.

Code

Python
import numpy as np

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

def analogy(E, vocab, a, b, c, k=3):
    """Return the k words closest to E[a] - E[b] + E[c]."""
    target = E[vocab[a]] - E[vocab[b]] + E[vocab[c]]
    sims = E @ target / (np.linalg.norm(E, axis=1) * np.linalg.norm(target))
    words = [w for w in sorted(vocab, key=lambda w: -sims[vocab[w]])
             if w not in (a, b, c)]
    return words[:k]

# 2-D projection with PCA
X = E - E.mean(axis=0)
_, _, Vt = np.linalg.svd(X, full_matrices=False)
coords = X @ Vt[:2].T

Practice problems

Problem 1: Compute a cosine similarity

Find the cosine similarity and the angle between x = (1, 2, 2) and y = (2, 0, 1).

Show solution

x · y = 1×2 + 2×0 + 2×1 = 4

|x| = √(1 + 4 + 4) = 3, |y| = √(4 + 0 + 1) = √5 ≈ 2.236

cos θ = 4 / (3 × 2.236) ≈ 0.596

cos θ ≈ 0.596, θ ≈ 53.4°

Problem 2: Solve an analogy

With dimensions (royalty, gender, person): king = (1, 1, 1), man = (0, 1, 1), woman = (0, −1, 1), queen = (1, −1, 1), girl = (0, −1, 0.8). Compute king − man + woman and decide whether queen or girl is the better answer.

Show solution

king − man + woman = (1−0+0, 1−1−1, 1−1+1) = (1, −1, 1)

cos(target, queen) = 3 / (√3 × √3) = 1.000

cos(target, girl) = (0 + 1 + 0.8) / (√3 × √1.64) = 1.8 / 2.218 ≈ 0.811

queen (cosine 1.0) is the answer

Discussion

Questions, ideas or a different approach? Join the conversation.