It would be easy to read this chapter as the ethics section at the end, to be agreed with and skipped.
That reading is available and it is wrong, for a technical reason.
Every harm described here traces to a mechanism established earlier in this book. Every measurement uses a tool the reader has already built.
Bias is not a moral failing bolted onto a neutral system. It is the distributional hypothesis of Chapter 6 working exactly as specified. Hallucination is not a bug. It is Equation (17.1) being minimised.
So the material is continuous with the rest, and it belongs at the end only because it needs everything before it.
Chapter 9 measured social regularities in a word embedding using WEAT, Equation (9.27). The finding was not that something had gone wrong. It was that a corpus of human text contains human associations, and a method built to learn associations learns those too.
That is the whole mechanism, and it does not change at scale. A larger model trained on more text learns the same associations more thoroughly.
The corpus. Web text over-represents some languages, regions, registers and viewpoints. Chapter 2 called this a sampling question, and it is the same question with higher stakes.
The labels. Annotation reflects the annotators. Who was hired, what they were told, and what they were paid all show up in the data.
The objective. Cross-entropy rewards fitting the training distribution. A model that reproduces a skew faithfully is scoring well.
The deployment. A model applied to a population unlike its training data will fail on that population specifically, and the aggregate metric will not show it.
The fourth is the one that survives the most careful modelling work, because it is not a property of the model at all.
Nothing here requires new machinery.
Association tests such as WEAT quantify how strongly two concept sets sit near two attribute sets. The score has a sign and a magnitude, and Chapter 9 showed what it does and does not license you to conclude.
Disaggregated evaluation is the most useful practice in this chapter. It is Chapter 14’s per-class table, applied to groups rather than labels.
Report your metric separately for each population you can identify.
An overall macro- of can hide on one group and on another. The aggregate is the number that gets reported.
Counterfactual testing changes one attribute in an input and asks whether the output changes. Swap a name, a pronoun, a dialect feature. If the prediction moves, the model is using that attribute.
All three are things you can run this week on a model you have.
Debiasing an embedding space by projecting out a direction reduces the measured score. It has repeatedly been shown to reduce the measurement more than the behaviour, because the information survives in other directions.
That is worth stating plainly. A metric that improves while the behaviour does not is worse than no metric.
This is Goodhart’s law from Chapter 21, arriving somewhere it does real harm.
Corpus curation, disaggregated reporting and deployment limits are slower and they address the source.
Three failure classes, each with a known cause and no complete fix.
A model trained to produce likely text produces plausible falsehoods, because a plausible falsehood is likely text. The practical responses are retrieval (Chapter 20) and a system that can say it does not know.
Small, deliberate perturbations change outputs. For text-driven systems the important case is prompt injection, and Chapter 22 argued that the defence is architectural rather than linguistic.
Performance degrades when live data drifts from training data. The response is monitoring rather than a better model.
None of these is solved. The systems that work in practice acknowledge them.
They express uncertainty, keep a human in the loop for consequential decisions, and fail safely.
Two practices, and both are cheap relative to what they prevent.
Model cards record intended use, training data, evaluation results including disaggregated ones, and known limitations.
Data statements record how a corpus was collected, from whom, with what consent, and what it does not cover.
Both exist because the alternative is a system whose limits are discovered by the people harmed by them.
Attention weights are often presented as explanations. They are not, and it is worth knowing why.
Attention says where information was read from. It does not say how that information was used, and models with very different attention patterns can produce identical outputs.
A chain-of-thought trace is a plausible narrative, not a record of the computation.
Chapter 18 gave the mechanism. The model is generating tokens that give it room to compute. Those tokens need not describe what the forward pass did.
Both are useful diagnostics. Neither is a guarantee, and presenting either as one is a transparency failure dressed as transparency.
A system’s behaviour is a shared responsibility. Model developers, deployers and operators each control something the others do not.
The one position that is not available is the one that blames the model. A model has no agency, so responsibility does not stop there. It stops with whoever chose to deploy it, for what, and with what checks.
This book followed one idea through twenty-odd chapters.
Language resists rules and yields to numbers and to learning. We counted words, turned counts into geometry, geometry into probability, and probability into models that converse.
At every stage the same three questions decided everything. What are we counting? What are we optimising? How do we know it worked?
Those questions do not go out of date, and they are what remains when the current architectures have been replaced.
Read a paper and identify its representation, its objective and its evaluation.
Build a baseline before reaching for a large model, and know why the baseline is often close enough.
Choose an evaluation before running an experiment, and hold to it when the result is disappointing.
Diagnose a failure by looking at the outputs rather than the score.
Say what a system cannot do, as precisely as what it can.
The last one is the hardest and the most valuable. A field that can state its limits accurately is a field that can be trusted with the things it does well.
is the WEAT paper, and a model of careful measurement. argue about the risks of scale and the consequences of corpus composition. propose model cards and data statements. treat ethics alongside the technical material throughout, which is the arrangement this book has tried to follow.
Trace a harm to a mechanism. Choose one failure from this chapter and trace it back to a specific equation or method earlier in the book. State the mechanism, not the sentiment.
Disaggregate. Take a classifier you have built. Split your test set by any attribute you can identify and report the metric per group. Compare the worst group against the aggregate. Would the aggregate have told you what you learned?
Counterfactual test. Construct twenty input pairs differing in exactly one attribute. Measure how often the prediction changes. State what a non-zero rate does and does not prove.
Debiasing that is not. Explain why projecting out a bias direction can reduce a WEAT score without changing behaviour. Design a test that would distinguish the two outcomes.
Write a model card. Produce one for a system you have built: intended use, training data, disaggregated evaluation, known limitations. The limitations section should be the longest one.
Attention is not explanation. Construct or describe two models with different attention patterns and identical outputs. What does this show about attention weights as an explanation, and what are they still good for?
Say what it cannot do. For a system you have built, write the paragraph you would give a user describing its limits. Then check each claim against an experiment you have actually run, and delete anything you cannot support.