Learning an Embedding Space

Part I — A Vector Is Not Meaning

Refusing the handout

For two chapters the vectors were given. Chapter 1 took them from an API and asked what was in them; Chapter 2 placed some by hand and treated the rest as the output of a model we agreed not to open. Chapter 2 closed on the question it left unanswered: where does the geometry come from?

This chapter builds a small space from nothing but a corpus and a counting rule. A usable geometry appears — related words land near each other without anyone labeling an axis “animal” or “drink.” Then comes the more important half: we inspect the construction closely enough to predict, from the mechanism alone, what it cannot preserve. When the measured results arrive, the failures are not surprises. They are consequences of the path the information took.

Where does the geometry in a learned embedding come from, and what operation produces it?

The short answer, which the rest of the chapter makes precise: the geometry is shaped by statistical regularities in the training data, filtered through an objective, a parameterization, and an optimization procedure. For the explicit method here that chain is short and every link is visible. For a modern encoder the chain is long, but the principle does not change.

Counting words in a tiny corpus

Six sentences:

S1  The cat drinks milk
S2  The dog drinks water
S3  The kitten drinks milk
S4  The puppy drinks water
S5  The cat chases the mouse
S6  The dog chases the cat

Set the aside for the toy example — it appears everywhere and makes the table harder to read. This is a pedagogical simplification, not exactly the preprocessing used in the measured RELATE run below, whose length filter keeps the.

Now build a within-sentence co-occurrence table. In these six sentences, each retained word appears at most once per sentence, so we can simply count how many sentences contain both words. The measured implementation generalizes this with a term-by-item count matrix M and computes M · Mᵀ, then zeros the diagonal. When terms repeat inside an item, that multiplication uses their occurrence counts rather than merely asking whether both are present. For this tiny corpus, the two procedures give the same off-diagonal table.

          cat  dog  kitten  puppy  milk  water  drinks  chases  mouse
cat        –    1     0       0      1     0      1       2       1
dog        1    –     0       0      0     1      1       1       0
kitten     0    0     –       0      1     0      1       0       0
puppy      0    0     0       –      0     1      1       0       0
milk       1    0     1       0      –     0      2       0       0
water      0    1     0       1      0     –      2       0       0
drinks     1    1     1       1      2     2      –       0       0
chases     2    1     0       0      0     0      0       –       1
mouse      1    0     0       0      0     0      0       1       –

Every entry is checkable by hand: cat and chases share S5 and S6, so their count is 2; milk and drinks share S1 and S3, so their count is 2.

Now read the rows as vectors. Three things stand out:

• cat and dog have similar rows. Both co-occur with drinks and with chases. Nothing in the table says they are both animals; they just fill the same slots.
• kitten and cat have similar rows even though their co-occurrence count is 0. They never appear in a sentence together. What they share is a context profile: both go with drinks and milk. Distributional similarity is similarity of contexts, not co-occurrence with each other.
• milk and water have similar rows and never co-occur. Same story: each pairs with drinks twice and with an animal once.

A row of this table is a distributional representation — a high-dimensional, sparse vector of contextual statistics. Mathematically it already lives in a continuous vector space such as ℝ^V; the fact that these particular coordinates happen to be integer counts does not make the surrounding vector space discrete. What it lacks is the learned, lower-dimensional transformation Chapter 1 used to distinguish an embedding from a hand-specified representation. Here the count rows are the explicit representation we will weight and factorize. That step is next.

From counts to a geometry

Two operations turn the count table into something you would actually query.

Weighting: PPMI. Raw counts over-reward frequency. drinks co-occurs with almost everything, so its large counts crowd out the fact that milk-with-drinks is more distinctive than cat-with-drinks. Positive pointwise mutual information rescales each entry by how surprising the co-occurrence is relative to the two words’ individual frequencies, and floors negative values at zero:

PPMI[a, b] = max(0,  log( count[a,b] · total  /  ( rowsum[a] · colsum[b] ) ))

After PPMI, a ubiquitous co-occurrence contributes less information than a pairing that occurs more often than its marginals would predict. Setting the aside earlier had a similar intent — stop one frequent token from dominating the toy — but it is not the same operation. Stopword removal deletes a feature; PPMI keeps the feature and reweights its association.

Factorization: truncated SVD. The PPMI matrix is still as wide as the vocabulary and mostly zeros. Factor it, PPMI ≈ U Σ Vᵀ, keep the top k components, and take each word’s row of U Σ as its k-dimensional vector. With k far below the vocabulary size, the result is a dense, lower-dimensional representation that preserves the strongest recurring structure available to a rank-k approximation. Correlated context profiles can therefore share directions instead of needing one coordinate per vocabulary item. The exact effect on a particular pair such as kitten and cat is something to measure after factorization, not something the toy counts alone guarantee.

This is a learned embedding in the operational sense: continuous, produced by fitting to data, with geometry meant to be read. Its structure was not handed down. It was compressed out of a counting rule.

Follow the information. A representation is a pipeline: preprocessing, then the representation function, then pooling, then the metric. Every stage can erase a distinction, and a distinction erased at any stage cannot be recovered by a later one. No similarity measure repairs information that preprocessing removed before the vector existed. Keep this in mind for the rest of the chapter.

The same principle, different machinery

The count-then-factor route above has close relatives. They are related historically and conceptually, but they are not one algorithm, and treating them as interchangeable hides differences that matter later.

Explicit factorization (PPMI-SVD, LSA). The route in this chapter. LSA is its older cousin: it applies SVD to a term-by-document matrix rather than the term-by-term PPMI matrix used here, so the object being factored differs, but the family move — build an explicit co-occurrence matrix, weight it, take a low-rank factorization — is shared.

GloVe. Also built from global co-occurrence counts, but it does not compute an SVD. It fits word and context vectors, together with bias terms, by weighted least squares so that wᵢ · w̃ⱼ + bᵢ + b̃ⱼ approximates log Xᵢⱼ. Its weighting function reduces the influence of very small counts and caps the weight of sufficiently frequent pairs. Same raw material, different objective and different optimizer.

Predictive embeddings (skip-gram, CBOW). Train a shallow model to predict a word from its neighbors, or the reverse, and keep learned word representations. This looks unrelated to counting. A well-known result (Levy and Goldberg, 2014) shows that one specific variant — skip-gram with negative sampling — corresponds, under particular assumptions, to factorizing a shifted PMI matrix, often summarized as PMI − log k. That is a precise connection for one objective, not a general identity between “prediction” and “counting.” It is enough to make the point: the two families are less separate than their code suggests.

Modern contextual encoders. Transformer-based representations may be trained with masked-token, next-token, contrastive, distillation, retrieval, or mixed objectives over large corpora. They are not simply low-rank factorizations of a co-occurrence matrix. Their objectives and architectures impose richer constraints, and — the part that changes the picture most — their outputs can depend on context, not just on the word type.

    flowchart TD
    C["corpus — text at scale"] --> M["statistical structure — which items share contexts, in what patterns"]
    M --> S1["explicit factorization — build a weighted co-occurrence matrix, take a truncated SVD (PPMI-SVD, LSA)"]
    M --> S2["global least squares — fit vectors so dot products track log co-occurrence (GloVe)"]
    M --> S3["predictive learning — predict a word from its context (skip-gram / CBOW)"]
    M --> S4["contextual encoder — masked / next-token / contrastive objectives over billions of tokens"]
    S1 --> E["a representation whose geometry exposes some of that structure"]
    S2 --> E
    S3 --> E
    S4 --> E
    S3 -.->|"SGNS optimum ≈ shifted-PMI factorization (Levy & Goldberg 2014)"| S1
  

What survives across all four is not an algorithm. It is a principle:

Structure in the training signal is converted into structure in a representation. What structure, and how accessible, depends on the objective, the parameterization, and the optimization — not on the word “embedding.”

What changes when the encoder is contextual

The count model gives bank exactly one vector. That vector is an average over every sentence bank appeared in — river banks, savings banks, blood banks — blended into a single point. The model represents a word type.

A contextual encoder represents something else. Run “the river bank collapsed” and “the bank approved the loan” through it and the vector sitting over bank differs between them, because it is computed from the surrounding tokens as well as the word itself. The object being represented is a token in context, or a span, or a whole sentence — not a type.

This is not the count model made larger. It is a different commitment about what gets a vector. A static word-vector space has one entry per vocabulary item; a contextual space has, in effect, one representation per occurrence. Later chapters lean on this distinction constantly, so it is worth stating plainly now: scaling a static model up does not make it contextual, and a contextual model is not a static model with more parameters.

What “learned compression” implies

Used carefully, “learned compression” is a good mental model. Three consequences of it hold for the explicit count model transparently, and for larger models with qualifications.

A vector is a summary of contexts, not a definition. milk sits near water because they occupy the same sentence slots, not because anything in the pipeline represents “liquid.” The geometry can behave as if it has a concept while being built entirely from distributional evidence.

Sparse evidence makes a vector hard to estimate. In the count model this is immediate: a word seen in three contexts has a vector compressed from three data points, and its position is noisy. Larger models soften this — subword pieces are shared across rare and common forms, and pretrained features transfer — so “rare word, bad vector” is not a universal law. The reliable version is: the less relevant evidence constrained a representation, the less you should trust its exact position. This returns as unstable neighborhoods (Chapter 6) and miscalibrated similarity (Chapter 14).

Associations in the data are candidates to become geometry. If the training text systematically links two concepts, that link can show up as proximity in the space — but whether it survives depends on the objective, sampling, architecture, regularization, and any later fine-tuning. The practical warning is therefore conditional rather than weaker: a space trained on web text can inherit web-text associations, including unwanted ones, when the learning process preserves them. A systematic association is a candidate to become geometric structure, not a guarantee that it will.

And a fourth point, stated to head off a misreading. Useful semantic behavior can emerge from statistical learning even though no coordinate is labeled with a concept. What emerges is relational structure that pays off under particular readouts and tasks — not “meaning” sitting inside the vector. Chapter 1 refused to say an embedding contains meaning, and building one from counts does not add any. It shows how far you can get without it.

Predict before you measure

Here is the homemade pipeline as an information path:

tokens(item)          = lowercase; keep runs of [a-z]; drop words of 2 letters or fewer
M[term, item]         = count of term in item
co                    = M · Mᵀ ,  diagonal zeroed              # term–term, within-item
PPMI[a, b]            = max(0, log( co[a,b] · total / ( rowsum[a] · colsum[b] ) ))
U, S, _               = svd(PPMI)
term_vector[t]        = (U · S)[t, :k]                         # k = 100
sentence_vector(item) = mean( term_vector[t] for t in tokens(item) ) , then L2-normalized
score(a, b)           = cosine( sentence_vector(a), sentence_vector(b) )

Before looking at any numbers, ask what this path has already thrown away by the time score runs:

1. Word order is gone. sentence_vector averages term vectors. Averaging is commutative, so any two sentences built from the same multiset of retained tokens produce the identical vector.
2. Syntax and compositional scope are gone. There is no structure that could represent what a not applies to, or which noun is the subject.
3. Digits are gone. [a-z] runs only. 1998 and 2012 are not low-weighted; they are deleted before counting.
4. Very short words are gone. The length > 2 filter drops no, of, to. It keeps not, never, fails.
5. Anything dropped here is unavailable downstream. No choice of k and no metric can bring it back.

From that list alone, three predictions:

• Order-only relation-swap pairs should score exactly 1.0. If two items reduce to the same multiset of retained tokens, mean pooling is commutative: they receive the same sentence vector regardless of k or of the learned term geometry.
• Many temporal-mismatch pairs should score very high. When the decisive difference is a year or quantity, the [a-z] tokenizer deletes that evidence before the term vectors exist. Any surviving non-numeric wording can still move the vector, so “near 1.0” is an expectation, not an algebraic guarantee.
• Minimal negations should remain very close to their source. Tokens such as not and never survive and can move the average, but the representation has no explicit syntax or scope saying that the token reverses the proposition. It is reasonable to expect high cosine. Whether the mean lands above paraphrase is an empirical result, not something token overlap alone mathematically guarantees.

Demonstration: grow the RELATE geometry from counts

MEASURED on RELATE v0.1, Wave 1 row 1.2 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/ppmi-svd-relate.json. PPMI term–term statistics over the 1,173-item corpus, SVD to 100 dimensions, vocabulary 1,349, sentence vector = mean of term vectors.

relation             mean cosine (PPMI-SVD-100)
equivalent                 0.98
relation-swap              1.0000   ← mean reported at four decimals
temporal-mismatch          0.9993   ← very nearly identical on average
negation                   0.95
partial-support            0.91
contradiction              0.89
paraphrase                 0.80
entailment                 0.74
topic-related              0.72
entity-related              0.64
unrelated                  0.28

The easy axis works. The mean of paraphrase, topic-related, and entity-related sits 0.44 above unrelated. Within-item co-occurrence genuinely captures aboutness: sentences about the same thing share terms, and the geometry places them together.

The assertion axis does not. contradiction (0.89) scores higher than paraphrase (0.80): the gap is −0.10, inverted relative to the neural encoder in Chapter 1 (+0.06). Paraphrases deliberately swap vocabulary while holding meaning fixed, so a token-overlap geometry reads them as less similar than contradictions that happen to reuse words.

The three predictions are borne out, but the mechanism needs to be stated at the level the experiment actually identifies.

• relation-swap has a reported mean of 1.0000. For any pair that reduces to the same retained-token multiset, mean pooling guarantees the same sentence vector: order has already vanished before cosine is computed. That is a pooling invariance, not a property the term geometry can repair. The relation-level artifact reports the mean rounded to four decimals, so the table alone does not justify the stronger claim that every pair is numerically identical.
• temporal-mismatch has a mean of 0.9993. The implementation deletes digits during preprocessing, so any distinction carried only by a year or quantity is literally unavailable downstream. Surviving words can still differ, which is why the mechanism predicts severe blindness to numeric changes without implying that every temporal pair must be identical.
• negation has a mean of 0.9476, above paraphrase at 0.7951. Tokens such as not and never can survive preprocessing and influence the average, but the averaging representation carries no explicit operator scope. The result is consistent with a representation that preserves lexical aboutness much more strongly than the logical effect of negation; the aggregate score does not isolate one single causal step as the whole explanation.

The useful lesson is not merely “bag-of-words is bad at assertion.” It is follow the information path: order is erased by pooling, digits can be erased by preprocessing, and compositional effects can be weakened by the representation itself. Similar-looking cosine failures need not have the same cause.

MEASURED: on these probes the homemade space shows the same broad shape as the neural encoders in Chapter 1 — relatedness separates well, assertion-sensitive distinctions do not. The shape is a family resemblance, not evidence that the neural models are scaled-up count models.

A counterexample worth keeping. On temporal-mismatch the homemade space scores 0.9993 while bge-large scores about 0.74. The comparison shows that the neural pipeline is substantially more responsive to this temporal-change probe. One plausible contributor is preprocessing: the homemade regex definitely deletes digits, whereas a subword tokenizer can preserve numerical information. But the two systems also differ in architecture, training objective, pooling, and learned geometry, so this experiment does not isolate the tokenizer as the sole cause. What it does establish is enough: the neural encoder is not merely a larger copy of the count pipeline.

What this chapter establishes and what it does not

Establishes: a corpus plus a counting rule plus a factorization yields a continuous space whose geometry exposes co-occurrence structure; explicit-count and predictive methods are formally connected for at least one predictive objective; a contextual encoder represents a different object (token/span/sentence in context) than a static word-vector model (word type); and, on the RELATE probes, the homemade space separates related from unrelated while inverting paraphrase versus contradiction.

Does not establish: that count-based and neural embeddings are interchangeable in practice (the temporal-mismatch gap shows they are not); that any specific relation is or is not captured by a given model (per-model, per-relation, empirical); or that the k = 100 choice here is good (Chapter 7 makes dimensionality a decision).

Lab 3: build your own space, then find its floor

MEASURED — artifact experiments/embeddings-from-first-principles/wave1/artifacts/ppmi-svd-relate.json. REPRODUCIBLE — run_wave1.py 1.2; the full implementation is the row_1_2 function in run_wave1.py, about fifty lines.

Question. What does a count-then-compress space get right for free, and which of its failures can no amount of tuning fix?

What we ran. A term–term PPMI matrix over RELATE (vocab 1,349), SVD to 100 dimensions, each sentence represented as the mean of its term vectors, scored on the same typed pairs as Lab 1 and compared with bge-large:

Interpretation. Establishes: the homemade space reproduces the neural encoder’s broad failure shape — aboutness easy, assertion hard — from a construction you can read end to end, and its sharpest failures trace to specific pipeline stages. Does not establish: a best k, or that the neural model differs from this one only in scale (the negation and temporal-mismatch rows say otherwise).

Try it yourself

Rebuild the space at k = 10, 50, and 200. First write down your prediction for each: what should changing k be able to improve, and what should it leave untouched? Then measure — five nearest neighbors for ten seed words, and the seven relation scores above.

The control that makes this experiment worth running is to separate an invariant from a hypothesis. For an order-only relation-swap pair with the same retained-token multiset, changing k cannot help: mean pooling has already made the two sentence representations identical. For temporal mismatch, changing k still cannot restore digits that preprocessing deleted, but surviving non-numeric differences can move as the factorization changes, so the score need not remain exactly pinned. As a final intervention, restore digits to the tokenizer ([a-z0-9]+) and re-check temporal-mismatch. That changes the information path itself rather than merely tuning a downstream dimension.

Companion component: the corpus fingerprint

If a space’s geometry is shaped by the data it was built from, the Observatory should keep a record of that data — and of the preprocessing, since this chapter has shown that a tokenizer decision can dominate a result.

corpus_fingerprint:
  n_items:              <int>
  vocab_size:           <int>
  token_count:          <int>
  length_distribution:  <summary>
  rare_item_fraction:   <items whose key terms have few contexts>
  duplicate_fraction:   <near-duplicates>
  domain_notes:         <dominant topics / registers>
  preprocessing:
    tokenizer:          <regex | subword model | ...>
    lowercased:         <bool>
    stopword_policy:    <none | list | min-length>
    number_handling:    <kept | normalized | dropped>
    context_definition: <window ±n | within-sentence | within-item>

These fields are not predictions of trouble. They are the conditions a later failure is most likely to depend on, recorded while they are still easy to obtain. When a similarity score looks wrong three chapters from now, this is the first place to look.

Failure modes

Each is a mistake, the reason it is tempting, and the check.

• “The model knows X.” Tempting because the geometry can behave as if it understands. Check: first ask whether the observed behavior can be explained by the training signal and representation mechanism. A distributional pattern can produce useful semantic behavior without establishing conceptual understanding.
• Trusting a sparse-evidence vector. Tempting because it has the same shape and type as every other vector. Check: for this count model, record how many contexts informed the term and test whether sparse terms have unstable neighborhoods. For a pretrained model, do not infer uncertainty from surface-form frequency alone; test the behavior you care about.
• Assuming a larger k is better. Tempting because more dimensions feel like more capacity. Check: in truncated SVD, increasing k improves reconstruction of the factored matrix, but that does not guarantee better neighbors, retrieval, or relation separation. Sweep k against the downstream measurement you actually care about (Chapter 7).
• Ignoring preprocessing. Tempting because it is upstream and invisible in the output vector. Check: read the tokenizer and the pooling step; whatever they discard is gone for good.
• Ignoring corpus provenance. Tempting because the space feels general. Check: name the corpus and its dominant registers before trusting a domain-specific judgment.

What this chapter established

• A corpus, a counting rule, PPMI weighting, and a truncated factorization produce a continuous space whose geometry was compressed out of observed counts rather than handed down.
• Explicit-count, global-least-squares, and predictive methods share one principle — training structure becomes representational structure — and are formally connected for at least one predictive objective. Related, not identical.
• A contextual encoder represents a token, span, or sentence in context; a static model represents a word type. The second does not become the first merely by getting larger.
• Follow the information path. Order can be destroyed by pooling, digits by preprocessing, compositional scope by an order-insensitive average — so similar-looking cosine failures need not share a cause. The corpus fingerprint records the preprocessing choices that decide which distinctions ever reach the representation at all.

Next

We now have spaces, handed to us and homemade, and a way to reason about what they discard. Every judgment so far — “close,” “near,” “more similar than” — has quietly assumed one way of comparing two vectors. The next chapter makes that assumption a decision: cosine, dot product, Euclidean, Manhattan, normalized or not. “Similar” turns out to be a choice before it is a measurement.