Chapter 15Contextual Embeddings

One vector is not enough

Part II built word vectors and Part III built models that use them, and both rest on an assumption nobody stated out loud, which is that every word gets exactly one vector, decided before any sentence exists.

Chapter 9 already showed the damage, since the bank of a river and the bank that holds money collapse into one point and that point is correct for neither sense. It is worth measuring rather than asserting.

What the average costs

Take a toy space on three axes for money, river and generic, then give the two senses the vectors a contextual model would produce.

$$\text{bank}_{\text{money}} = (0.90,\, 0.05,\, 0.20), \qquad \text{bank}_{\text{river}} = (0.05,\, 0.90,\, 0.20).$$

A static embedding must pick one vector, and trained on a corpus carrying both senses equally it lands on their average, $(0.475, 0.475, 0.200)$. Now ask all three for nearest neighbours.

The two sense vectors give clean, single-topic lists, while the average gives a list that mixes both and commits to neither.

Measure the geometry directly. The static average sits at cosine $0.7591$ from each sense, which sounds close until you notice the two senses are only $0.1525$ from each other. The static vector is therefore equally wrong in two directions rather than right in one.

It gets worse, or better, with the corpus

The average is not a fixed compromise either, because it moves with whatever mixture the training corpus happened to contain.

At $90$ per cent the vector is almost entirely the money sense and the river sense has been erased.

That is the honest statement of what a static embedding holds. It does not represent a word but that word’s frequency-weighted mixture in one particular corpus. Chapter 9 made the same point with apple, and here it is as arithmetic.

Embeddings from a language model

The repair is stated in one sentence. Do not look the vector up but compute it, from the sentence the word is actually in. ELMo (Peters et al. 2018) was the model that made this practical, and its name is the method, Embeddings from Language Models.

The architecture

Three pieces make it up, and each earns its place.

The input is character-based, so the bottom layer builds each word’s initial representation from its characters through a convolution. ELMo therefore has no vocabulary to fall off, and a word it has never seen still gets a sensible vector, which is the FastText argument of Chapter 9 applied at the input layer.

Above that sits a bidirectional language model, where two multi-layer LSTMs run over the sentence, one forward and one backward, the forward one predicting the next token and the backward one the previous. Both are ordinary language models in the sense of Chapter 10, trained by maximum likelihood on plain text.

The third piece is the layer-wise combination, and it is the part worth dwelling on.

The layer mix

A biLM with $L$ layers gives $L+1$ representations for each token, being the character layer plus one per LSTM layer, and ELMo does not pick one but combines them all:

$$\begin{equation} \text{ELMo}_k \;=\; \gamma \sum_{j=0}^{L} s_j\,R_{k,j}, \label{eq:elmo} \quad\text{(15.1)} \end{equation}$$

where the $s_j$ are softmax-normalised weights and $\gamma$ is a single scaling factor.

The detail that matters is which parameters are learned downstream, and the answer is only the $s_j$ and $\gamma$, with the biLM itself frozen. That is a handful of numbers per task, which is why ELMo was cheap enough to adopt everywhere within months.

The weights also turn out to be readable, because different tasks learn different profiles.

Tagging leans on the lower layer while sense disambiguation leans on the upper one, and that is the finding the ELMo paper is remembered for: lower layers carry more syntax and upper layers more meaning. Which one a task wants is something you read off the learned weights rather than decide in advance, and it is also the clearest evidence that depth in these models does something structured rather than merely adding capacity.

One more piece of machinery

Deep stacks need their activations kept in range, and the lectures introduce the tool here because everything after uses it. Layer normalisation rescales each vector to zero mean and unit variance, then applies a learned scale and shift:

$$\begin{equation} \operatorname{LayerNorm}(x) \;=\; \gamma \cdot \frac{x - \mu}{\sigma} \;+\; \beta , \label{eq:layernorm} \quad\text{(15.2)} \end{equation}$$

where $\mu$ and $\sigma$ are the mean and standard deviation of $x$ itself.

The contrast with batch normalisation is the point. Batch norm computes its statistics across the batch, so its behaviour depends on what else is in the batch and on the batch size, whereas layer norm computes them within a single example. That makes layer norm independent of batch composition, which matters when examples have different lengths, and it also behaves identically at training and inference time. Sequence models have exactly that property, which is why layer norm is universal in transformers and batch norm is not.

Pretrain once, adapt cheaply

ELMo established a pattern that outlived the architecture. Train a large model once on plain text using a self-supervised objective that needs no labels, then adapt it to each task with a small amount of labelled data.

The economics are the argument, since pretraining is expensive and happens once while adaptation is cheap and happens many times. For ELMo the adaptation was minimal, being just the $L+1$ weights of Equation (15.1) and one scalar, and later models fine-tune more without changing the shape of the bargain. Everything in Parts IV and V is a variation on it.

What it bought, measurably

ELMo improved the state of the art on six tasks at once, and that breadth is unusual enough to be worth naming: question answering, textual entailment, sentiment analysis, named entity recognition, coreference resolution and semantic role labelling. One architecture change, six benchmarks and no task-specific engineering, which is what convinced the field more than any single number did.

The mechanism behind it is the one this chapter opened with, namely that a word’s representation now depends on its sentence, so bank in “the river bank” and bank in “the savings bank” are different vectors rather than one blurred average.

The idea outgrows its machine

ELMo proved the principle and inherited a limitation. Its two LSTMs are the recurrent networks of Chapter 13, with Chapter 12’s two burdens intact, so they process one token at a time and carry everything through a fixed size state.

There is a subtler point too, which is that the forward and backward models are independent and only combined at the end. No layer of ELMo ever sees left and right context at once, and what it sees instead is two one-sided summaries stitched together.

Chapter 16 removed the recurrence and Chapter 18 puts the two together, giving a model that reads both directions in every layer at once.

Try it yourself. code/worked_examples/contextual.py produces the tables in this chapter. --bank measures what averaging two senses costs and how it shifts with the corpus mixture, --elmo prints the three task profiles, and --mask previews the masking budget of Chapter 18.

Run it in ColabNotebookSource

Further reading.

Peters et al. (2018) is the ELMo paper, and its analysis of what each layer holds is the part worth reading closely. Ba et al. (2016) introduce layer normalisation. Jurafsky and Martin (2025) cover contextual representations alongside the static ones, so the contrast is easy to follow. Any of these models is three lines with transformers, where AutoTokenizer and AutoModel will fetch a pretrained checkpoint and hand you the per-token vectors this chapter describes.

Code for this chapter

Each one reproduces the tables above. Open it in Colab, change an input and watch which numbers move.

• contextual.pyWhat one vector per word costs

  Averaging two senses into one vector, and the layer weights ELMo learns instead.

  standard library only Run it in Colab Notebook Source

From Natural Language Processing: From Corpus Statistics to Grounded LLMs, by Ramaseshan Ramachandran.