Chapter 15Contextual Embeddings

One vector is not enough

Part II built word vectors and Part III built models that use them. Both rest on an assumption nobody stated out loud.

Every word gets exactly one vector, decided before any sentence exists.

Chapter 9 already showed the damage. The bank of a river and the bank that holds money collapse into one point, and that point is correct for neither sense.

It is worth measuring rather than asserting.

What the average costs

Take a toy space on three axes, money, river and generic. Give the two senses the vectors a contextual model would produce.

$$\text{bank}_{\text{money}} = (0.90,\, 0.05,\, 0.20), \qquad \text{bank}_{\text{river}} = (0.05,\, 0.90,\, 0.20).$$

A static embedding must pick one vector. Trained on a corpus carrying both senses equally, it lands on their average, $(0.475, 0.475, 0.200)$.

Now ask all three for nearest neighbours.

The two sense vectors give clean, single-topic lists. The average gives a list that mixes both and commits to neither.

Measure the geometry directly. The static average sits at cosine $0.7591$ from each sense. That sounds close, until you notice the two senses are only $0.1525$ from each other.

So the static vector is equally wrong in two directions rather than right in one.

It gets worse, or better, with the corpus

The average is not a fixed compromise. It moves with whatever mixture the training corpus happened to contain.

At $90$ per cent the vector is almost entirely the money sense. The river sense has been erased.

That is the honest statement of what a static embedding holds. It does not represent a word. It represents that word’s frequency-weighted mixture in one particular corpus.

Chapter 9 made the same point with apple. Here it is as arithmetic.

Embeddings from a language model

The repair is stated in one sentence. Do not look the vector up. Compute it, from the sentence the word is actually in.

ELMo was the model that made this practical, and its name is the method: Embeddings from Language Models.

The architecture

Three pieces, and each earns its place.

Character-based input.

The bottom layer builds each word’s initial representation from its characters, through a convolution. So ELMo has no vocabulary to fall off. A word it has never seen still gets a sensible vector. That is the FastText argument of Chapter 9, applied at the input layer.

A bidirectional language model.

Two multi-layer LSTMs run over the sentence, one forward and one backward. The forward one predicts the next token, the backward one the previous. Both are ordinary language models in the sense of Chapter 10, trained by maximum likelihood on plain text.

Layer-wise combination.

This is the part worth dwelling on.

The layer mix

A biLM with $L$ layers gives $L+1$ representations for each token: the character layer, plus one per LSTM layer. ELMo does not pick one. It combines them:

$$\begin{equation} \text{ELMo}_k \;=\; \gamma \sum_{j=0}^{L} s_j\,R_{k,j}, \label{eq:elmo} \quad\text{(15.1)} \end{equation}$$

where the $s_j$ are softmax-normalised weights and $\gamma$ is a single scaling factor.

The detail that matters is which parameters are learned downstream. Only the $s_j$ and $\gamma$. The biLM itself is frozen.

That is a handful of numbers per task, which is why ELMo was cheap enough to adopt everywhere within months.

The weights also turn out to be readable. Different tasks learn different profiles.

Tagging leans on the lower layer. Sense disambiguation leans on the upper one.

That is the finding the ELMo paper is remembered for. Lower layers carry more syntax, upper layers more meaning.

Which one a task wants is something you read off the learned weights, rather than decide in advance.

It is also the clearest evidence that depth in these models does something structured. Depth is not merely added capacity.

One more piece of machinery

Deep stacks need their activations kept in range, and the lectures introduce the tool here because everything after uses it.

Layer normalisation rescales each vector to zero mean and unit variance, then applies a learned scale and shift:

$$\begin{equation} \operatorname{LayerNorm}(x) \;=\; \gamma \cdot \frac{x - \mu}{\sigma} \;+\; \beta , \label{eq:layernorm} \quad\text{(15.2)} \end{equation}$$

where $\mu$ and $\sigma$ are the mean and standard deviation of $x$ itself.

The contrast with batch normalisation is the point. Batch norm computes its statistics across the batch. So its behaviour depends on what else is in the batch, and on the batch size.

Layer norm computes them within a single example. That makes it independent of batch composition, which matters when examples have different lengths. It also behaves identically at training and inference time.

Sequence models have exactly that property, which is why layer norm is universal in transformers and batch norm is not.

Pretrain once, adapt cheaply

ELMo established a pattern that outlived the architecture.

Train a large model once on plain text, using a self-supervised objective that needs no labels. Then adapt it to each task with a small amount of labelled data.

The economics are the argument. Pretraining is expensive and happens once. Adaptation is cheap and happens many times.

For ELMo the adaptation was minimal, just the $L+1$ weights of Equation (15.1) and one scalar. Later models fine-tune more, but the shape of the bargain has not changed.

Everything in Parts IV and V is a variation on it.

What it bought, measurably

ELMo improved the state of the art on six tasks at once. That breadth is unusual enough to be worth naming.

Question answering, textual entailment, sentiment analysis, named entity recognition, coreference resolution and semantic role labelling.

One architecture change, six benchmarks, no task-specific engineering. That breadth is what convinced the field, more than any single number.

The mechanism behind it is the one this chapter opened with. A word’s representation now depends on its sentence.

So bank in “the river bank” and bank in “the savings bank” are different vectors, not one blurred average.

The idea outgrows its machine

ELMo proved the principle and inherited a limitation.

Its two LSTMs are the recurrent networks of Chapter 13, with Chapter 12’s two burdens intact. They process one token at a time, and they carry everything through a fixed-size state.

There is a subtler point too. The forward and backward models are independent and only combined at the end. So no layer of ELMo ever sees left and right context at once. It sees two one-sided summaries, stitched together.

Chapter 16 removed the recurrence. Chapter 17 puts the two together, and the model that results reads both directions in every layer at once.

Try it yourself. code/worked_examples/contextual.py produces the tables in this chapter. --bank measures what averaging two senses costs and how it shifts with the corpus mixture, --elmo prints the three task profiles, and --mask previews the masking budget of Chapter 17.

Run it in ColabNotebookSource

Further reading.

is the ELMo paper, and its analysis of what each layer holds is the part worth reading closely. introduce layer normalisation. cover contextual representations alongside the static ones, so the contrast is easy to follow.