A pretrained model knows what was in its training corpus, frozen at the moment training stopped. Three kinds of question therefore have no good answer.
Recent facts. Anything after the cutoff date does not exist for the model.
Private facts. Your company’s policies, your codebase, your customer records. None of it was in the corpus, and none of it should have been.
Rare facts. Things technically present in the corpus but seen so few times that the model’s memory of them is unreliable.
In all three cases the model does not say it does not know. It produces the most plausible continuation, which is a confident and specific falsehood.
That is Chapter 19’s hallucination, and no amount of prompting fixes it, because prompting selects from what the model has.
Do not ask the model to remember. Give it the document and ask it to read.
That is retrieval-augmented generation . Find the relevant text, put it in the context window, and generate an answer grounded in it.
The change in what is being asked is the whole point. Recall from parameters is unreliable and unverifiable. Reading comprehension over supplied text is something these models are extremely good at, and the source can be shown.
Four stages, and almost every part is machinery from Part II.
Documents are too long for a context window, and too long to retrieve usefully. So split them.
Chunk size is a real trade-off, not a detail.
| chunk size | retrieval | context supplied |
|---|---|---|
| too small | precise matches | fragments, missing sense |
| too large | one vector, many topics | whole sections, mostly noise |
The failure at the small end is easy to miss. A chunk of one sentence retrieves precisely and then hands the model a sentence whose antecedents are in the paragraph above.
Overlapping chunks reduce that, at the cost of storing the overlap. Splitting on structure, at paragraph or section boundaries, usually beats splitting at a fixed token count.
Each chunk becomes a vector, from an embedding model. This is Chapter 15’s machinery, applied to passages rather than words.
The vectors go into an index built for approximate nearest-neighbour search. Exact search over millions of vectors is too slow, and approximate search gives up a small amount of recall for a very large amount of speed.
Embed the query the same way and find the nearest chunks by cosine similarity, which is Equation (9.26) from Chapter 9.
Dense retrieval of this kind has one clear advantage over keyword search and one clear weakness.
It matches on meaning, so a query about “refund window” can retrieve a passage saying “returns accepted within 30 days”, with no shared words.
It is weak on exact strings. A part number, a person’s name, an error code. Those are precisely the cases where a keyword index is unbeatable.
So production systems usually run both and merge, which is called hybrid retrieval. The two methods fail on different queries, which is the only good reason to run two of anything.
Put the retrieved chunks in the prompt with the question and ask for an answer based on them.
The instruction that matters most is the one about not knowing.
Answer using only the passages below. If they do not contain the answer, say so.
Without that line the model falls back on its parameters and you are back to hallucination, now with citations that look authoritative.
With it, the system can return “I don’t know”, which is the single most valuable behaviour a retrieval system has.
| problem | does RAG help? |
|---|---|
| model lacks recent or private knowledge | yes, directly |
| answers cannot be verified | yes, the source is shown |
| knowledge changes often | yes, update the index not the model |
| model reasons badly over supplied text | no |
| model’s default behaviour is wrong | no, that is Chapter 19 |
| the answer is not in any document | no, and it should say so |
The third row is the underrated one. Updating a document store is a database write. Updating a model’s parametric knowledge is a training run.
The fourth row is the honest limitation. RAG supplies context. It does nothing for a model that misreads the context it is given.
Almost every RAG failure is a retrieval failure, and diagnosing it takes two steps rather than one.
First ask whether the right chunk was retrieved at all. If it was not, the generator never had a chance and no prompt engineering will help.
Only if the right chunk was retrieved and the answer is still wrong is the generator at fault.
That split matters because the two have entirely different repairs. Retrieval failures are fixed with chunking, embeddings, hybrid search or reranking. Generation failures are fixed with prompting or a better model.
Teams routinely spend weeks on the second when the problem was the first.
introduced the term and the architecture. is the dense passage retrieval paper that made the retrieval half work. covers the information retrieval foundations, including the keyword methods that hybrid search still depends on.
Chunk it wrong, twice. Take a document and chunk it at tokens and at . For a handful of questions, record which chunks are retrieved. Describe the failure mode at each extreme in terms of what the generator receives.
Dense against keyword. Write five queries where dense retrieval should beat keyword search, and five where it should lose. State the property that separates the two lists.
The refusal instruction. Build a small RAG system, then ask it a question whose answer is not in any document. Report what it says with and without the instruction to refuse. Which behaviour would you ship?
Diagnose the layer. A RAG system gives a wrong answer. Describe the exact procedure you would follow to decide whether retrieval or generation is at fault, and name the repair you would attempt in each case.
Update cost. A policy document changes weekly. Compare the cost of keeping a RAG index current against the cost of keeping a fine-tuned model’s knowledge current. Then name one situation where fine-tuning is still the right answer.