Translation has a target, in that two competent translators produce texts that are recognisably the same. Summarisation does not work that way, because two competent people summarising the same article will disagree about what mattered and both can be right.
That makes it the hardest generation task to evaluate and the most useful one to study, since every difficulty in Part V shows up here at once: decoding choices, faithfulness, evaluation that does not measure what you care about, and a context window that is too small.
The word covers two quite different problems, and they are worth separating.
| extractive | abstractive | |
|---|---|---|
| what it does | selects sentences from the source | generates new text |
| can it hallucinate? | no, by construction | yes |
| how it reads | disjointed | fluent |
| what it is | a ranking problem | a generation problem |
Extractive summarisation cannot state anything the source did not, which is a guarantee no generative system can offer, and it reads badly because sentences lifted out of context do not join up. Abstractive summarisation reads well and can compress genuinely, saying in one clause what the source took a paragraph to say, and it can also state things the source never said. Transformers made abstractive summarisation practical and made faithfulness the central problem, and those two facts are the same fact.
This is the part most often skipped and it changes everything downstream, because two datasets dominate the literature and they are not the same task. CNN/DailyMail pairs news articles with their bullet-point highlights, and those highlights are largely extractive, since most of their content appears near-verbatim in the first few sentences of the article. XSum pairs BBC articles with their one-sentence opening summary, which is written to be genuinely abstractive and often states something no single sentence of the article states.
Three consequences follow. A model tuned on one transfers poorly to the other, because they reward different behaviour. On CNN/DailyMail a lead-sentence baseline, which simply returns the first few sentences, is competitive, so if your model cannot beat it then you have not established that it learned anything. And hallucination rates on XSum are far higher, which is not a defect of the models, because the dataset asks for statements not present in the source and a model that obliges will sometimes oblige incorrectly.
The practical rule is short. Before choosing a model, measure how extractive your target summaries are, because that number tells you what you are actually training.
Two shapes are available and both are in Chapter 18. An encoder-decoder reads the document with a bidirectional encoder and writes with a causal decoder, so the encoder sees the whole source at once, which is the natural fit and what the architecture was designed for. A decoder-only model puts the document and the summary in one sequence and continues it, which is what a general instruction-tuned model does when you ask it to summarise.
The encoder-decoder shape has the better inductive bias and needs fine-tuning, while the decoder-only shape needs no fine-tuning at all and arrives already competent, which usually wins on engineering grounds.
Pretraining objective matters more than shape, though, since models pretrained by reconstructing corrupted spans of text rather than by predicting the next token transfer to summarisation better, because span reconstruction is the same operation summarisation performs.
Chapter 16 established that attention costs in the sequence length, and summarisation is where that bill arrives. A news article fits in a few hundred tokens, a scientific paper is five to ten thousand, a transcript or a legal filing is longer still, and many checkpoints cap at 512 or 1024.
There are three responses, each with an honest cost. Truncation is cheap and silently discards the tail of the document, which on news is nearly free because news is written with the important part first, and on a paper removes the results. Sparse attention, as in Longformer, LED and BigBird, has each position attend to a local window plus a few global positions, which brings the cost back to roughly linear and buys length at some cost in quality. And hierarchical summarisation summarises chunks and then summarises the summaries, which handles any length and compounds errors, because a mistake in a chunk summary is invisible at the next level.
Measure the length distribution of your inputs and your targets before you set the truncation limits. Truncating inputs is visible, since you know you did it, whereas truncating targets is silent and it is a common bug.
A target cut at the limit teaches the model to stop early, and the loss looks fine throughout training because the model is correctly predicting a truncated target. The failure only appears at generation time, as summaries that end mid-thought.
Training is the cross-entropy of Chapter 11, with two details that matter. Mask the loss, because padding tokens must not contribute, and an unmasked loss over a heavily padded batch is mostly measuring the model’s ability to predict padding, which will look encouragingly low. And teacher forcing feeds the decoder the correct previous token at each step rather than its own prediction, which makes training parallel and stable and introduces the problem below.
The mismatch is exact. During training every previous token the model sees is correct, while during generation the model consumes its own output, so the model has never been trained on the distribution it faces at inference and one early error shifts the context for everything after it. The model has no experience of recovering from its own mistakes, because in training it never made any.
Three partial remedies exist and none is a cure. Label smoothing stops the model becoming over-confident, which makes it less brittle when the context is unfamiliar, scheduled sampling occasionally feeds the model’s own predictions during training, and sequence-level fine-tuning optimises against the evaluation metric rather than per-token likelihood.
Exposure bias is also the cleanest reason decoding matters here, since sampling at high temperature compounds it, because an unlikely token early is a corrupted context for everything that follows. Summarisation wants the low-entropy end of Chapter 19’s table.
Chapter 22 covered ROUGE, which is the standard here, and two things matter about using it.
Evaluate with generation rather than loss, because validation loss measures next-token prediction under teacher forcing, which is not the task, so generate summaries and score them. That is slower and it is the number that matters, since watching loss alone means watching a quantity that improves while the thing you care about does not, and Chapter 22 named that failure.
ROUGE rewards overlap, and overlap is not faithfulness, so a summary can score well and state something false while a correct paraphrase can score near zero, exactly as Chapter 22 showed with the feline rests upon the rug.
For summarisation the gap is specific and it has a name. Faithfulness is whether every claim in the summary is supported by the source, and ROUGE does not measure it, cannot measure it, and a system tuned on ROUGE alone will drift away from it.
Measuring faithfulness needs a different instrument, whether entailment models asking whether the source entails each summary sentence, question-answering consistency checks, or human annotation. All three are more expensive than ROUGE, which is why ROUGE is still reported.
The chapter has assumed you are training a model, and often you should not. A general instruction-tuned model summarises well with no training at all, which is Chapter 19’s point, and fine-tuning earns its cost when you need a specific style, a specific length, a specific domain vocabulary or an on-premises model.
When you do fine-tune, Chapter 24 applies unchanged, where LoRA at rank 8 trains four tenths of one per cent of the weights, which is usually enough for a task this close to what the model already does.
The order to work in is Chapter 19’s hierarchy. Prompt first, measure, and fine-tune only when you can say what prompting failed to achieve.
Lin (2004) is ROUGE, and its limitations are best read alongside Zhang et al. (2020). Lewis et al. (2020) introduced the span-corruption pretraining that transfers so well here. Jurafsky and Martin (2025) cover both extractive and abstractive summarisation with more task detail.
Measure your dataset. For a summarisation corpus of your choice, compute what fraction of summary bigrams appear verbatim in the source. Compare it against a lead-sentence baseline’s ROUGE. What does the gap tell you about what a trained model would have to learn?
Beat the lead baseline. Implement the baseline that returns the first three sentences and score it. Then argue what a model must do to justify its cost, and state the margin you would require.
The silent truncation. Explain why truncating targets produces a training loss that looks healthy while generation fails. Describe the check you would add to a training script to make the bug loud.
Exposure bias, concretely. Trace what happens when a decoder generates one wrong token at position three of a twenty-token summary. Explain why teacher forcing gives the model no experience of this, and say which of the three remedies addresses it most directly.
Two metrics disagree. Construct a summary that scores well on ROUGE and is unfaithful to its source, and one that is faithful and scores poorly. Then describe an evaluation that separates them.
Choose the architecture. You must summarise ten-thousand-token legal filings, on-premises, in a fixed house style. Work through the choices in this chapter: extractive or abstractive, encoder-decoder or decoder-only, truncate or sparse or hierarchical, prompt or fine-tune. Justify each, and name the one you are least sure about.
From Natural Language Processing: From Corpus Statistics to Grounded LLMs, by Ramaseshan Ramachandran.