Chapter 18Pretrained Transformers: BERT and GPT

Two doors from one building block

Chapter 16 built a block, and this chapter builds the two families that came out of it. The only real difference between them is what each one is allowed to see.

Take the attention of Equation (16.2), apply the causal mask of Equation (16.3) and each position sees only its past, whereas leaving the mask off lets each position see the whole sentence. That single choice is the fork between GPT and BERT and most of what follows is downstream of it, although the table below shows the two also differ in the objective they train against.

The last row is not a limitation anyone chose, because it falls out of the second row, and Section 18.3.3 explains why.

GPT: pretraining by prediction

GPT (Radford et al. 2018) is a stack of decoder blocks trained on the oldest objective in this book, which is to predict the next token given the tokens so far. That is Equation (10.1) from Chapter 10, with a transformer in place of the count table:

$$\begin{equation} \mathcal{L} \;=\; -\sum_i \log p\big(w_i \mid w_1, \ldots, w_{i-1}\big). \label{eq:gpt-loss} \quad\text{(18.1)} \end{equation}$$

Nothing in that equation is new, since the n-gram model of Chapter 10 minimised the same quantity and so did the recurrent language model of Chapter 12. What changed is the machinery computing the probability, and how much data it could be fed.

Why the causal mask is not optional here

Without the mask, position $i$ could attend to position $i+1$ and read the answer it is being asked to predict, so the loss would fall to nearly zero and the model would learn nothing. This is the reasoning that made </s> a scored token in Chapter 10, because an objective that can be satisfied trivially is the wrong objective. The mask is therefore what makes the task hard enough to be worth solving.

What prediction turns out to teach

The surprise of the GPT line is how much falls out of so little. To predict the next token well across a large corpus a model has to represent a great deal, including syntax, facts, discourse structure and something that functions like reasoning. None of that was asked for, and all of it is instrumental to the one objective.

That claim was made repeatedly and doubted repeatedly, and the evidence for it is in Chapter 19, where models trained this way answer questions they were never trained on.

BERT: pretraining by filling gaps

BERT (Devlin et al. 2019) drops the mask and therefore needs a different objective, because if every position can see every other then next-token prediction is trivial. So BERT hides tokens instead and predicts those, which is masked language modelling, and it is the cloze task from language teaching.

The masking budget

Two numbers define it, and both are trades worth seeing. The first is how much to hide, so take a sequence of $512$ tokens.

At one per cent you pay for a full forward pass over $512$ tokens and collect five predictions, which is correct and very slow. At ninety per cent you collect plenty of predictions with almost no context to make them from, so the task degenerates into guessing.

BERT chose $15$ per cent, which gives $76$ gradients per sequence with $85$ per cent of the sentence still readable. Later work has found higher rates workable on larger models, so this is a balance point rather than a constant.

The mismatch, and the fix

The second number is subtler, and it repairs a problem the first one creates. The token [MASK] appears in every pretraining batch and never once at fine-tuning time, so a model that keyed on it would break the moment it was used for real. Of the $15$ per cent chosen, BERT therefore does three different things.

The last two rows carry the argument. Because a chosen position might be left unchanged, the model cannot tell which positions were chosen, so it has to build a usable representation of every token rather than only the obviously missing ones.

The random-token row teaches something else again, since it forces the model to notice when a token does not fit its context, and no amount of [MASK] prediction would ever require that.

Why BERT cannot generate

Generation means producing token $i$ from tokens $1$ to $i-1$, then $i+1$ from $1$ to $i$, and so on down the sequence. Every layer of BERT attends in both directions, so position $i$ has been built from the whole sequence, including positions that do not exist yet during generation.

You could mask BERT at inference time, and it would then be used in a configuration it was never trained in, where it performs badly. This is the same restriction the bidirectional RNN had in Chapter 12, for the same reason: reading both ways and generating left to right are incompatible requirements, and a model has to choose.

The same objective, scaled

The GPT line’s later chapters are mostly one experiment repeated with more of everything in it.

The architecture barely changed across that table and the objective did not change at all, so what changed was scale, and something unexpected came with it. At GPT-3’s size the model could perform tasks from a description and a few examples in its prompt, with no gradient update at all (Brown et al. 2020).

That is in-context learning and nobody designed it. It is the behaviour that turned language models from a component into a product, and Chapter 19 is about using it.

Where the parameters go

The counting of Chapter 16 scales up unchanged. For a transformer with model dimension $D$, $L$ layers and feed-forward size $H = 4D$, each layer costs roughly $12D^2$, which is $4D^2$ for the four attention projections and $8D^2$ for the two feed-forward ones.

The quadratic in $D$ is why width is expensive and the linear in $L$ is why depth is comparatively cheap. Both are dwarfed by the third row, where each of the $96$ layers costs $1.8$ billion parameters on its own.

What a pretrained model is for

There are two ways of using one, and the field has moved from the first to the second. Fine-tuning continues training on a labelled task, updating all the weights, which is what BERT was designed for, and it works well with a few thousand examples while producing one model per task. Prompting leaves the weights alone and describes the task in the input, so it needs no labelled data and no training run, and one model serves every task.

Fine-tuning generally scores higher when you have the labels, and prompting wins on everything else, while Chapter 24 shows how to get most of fine-tuning’s benefit at a fraction of its cost.

The deeper shift is in what a model is. Before pretraining a model was built for a task, and after it a model is a general capability that tasks are requested from. Part V is about making those requests well, checking the answers, and living with what the model learned from its corpus.

Further reading.

The first GPT paper is Radford et al. (2018), and Brown et al. (2020) reported in-context learning. For BERT itself go to Devlin et al. (2019), whose ablations on the masking scheme are worth the read on their own. Vaswani et al. (2017) remains the architecture both are built from. Jurafsky and Martin (2025) cover the pretraining and fine-tuning pattern with more task detail. Both families are served by transformers, where the choice between them is the difference between AutoModelForMaskedLM and AutoModelForCausalLM, which is the whole of this chapter in two class names.

Code for this chapter

Each one reproduces the tables above. Open it in Colab, change an input and watch which numbers move.

• contextual.pyWhat one vector per word costs

  Averaging two senses into one vector, and the layer weights ELMo learns instead.

  standard library only Run it in Colab Notebook Source

From Natural Language Processing: From Corpus Statistics to Grounded LLMs, by Ramaseshan Ramachandran.