Retrieval-augmented generation is the least glamorous idea in applied AI and the one that carries most of the weight. The mechanism is simple: index your documents, find the passages that relate to a question, put those passages in front of the model, and ask it to answer from them. Everything interesting is in what that does and does not buy you.
What it genuinely fixes
- Currency. The model does not need to have been trained on your handbook. Update the file and the next answer reflects it.
- Attribution. Because the answer is built from retrieved passages, the system can show which ones. An answer you can trace is an answer you can check.
- Exposure. Only the relevant passages are sent, not the file. A hundred-page agreement contributes three paragraphs to the prompt.
- Scale. Nobody has read all of it. Retrieval is indifferent to whether the corpus is fifty documents or fifty thousand.
That last one is the reason it earns its place. The value is rarely a cleverer answer; it is an answer at all, from material no individual has the time to hold in their head.
What it does not fix
It does not make the model reason better. If the question requires synthesis across twenty documents with conflicting positions, retrieval will surface the conflict but not resolve it, and a confident summary that flattens the disagreement is worse than no summary.
It does not fix your documents. If the policy is contradicted by a newer memo that was never filed, retrieval returns the policy, correctly and uselessly. This is the most common cause of a disappointing pilot, and it is not a technology problem.
Retrieval inherits the quality of the corpus. It makes a good archive searchable and a bad archive confidently wrong.
It also does not remove the need to check. Retrieval reduces invention; it does not eliminate it. A model handed three passages can still produce a fourth claim that reads exactly like the others.
Where implementations go wrong
Chunking that cuts through meaning
Documents are split before indexing. Split them by character count and you will eventually cut a table in half, or separate a clause from the definition it depends on. The retrieved passage looks fine and means something different from the document.
Retrieval that only understands similarity
Semantic search finds passages that resemble the question. That fails on exact identifiers — a contract number, a part code, a person's name — where you needed a literal match. Systems that combine keyword and semantic retrieval avoid a category of frustrating near-misses.
Permissions applied after the fact
If the index is built once for everyone and filtered at display time, then the model has already seen material the asker was not entitled to, and a well-phrased question can extract it. Permissions belong in the retrieval step.
Retrieval that stops at the first hit
Many questions need several passages from different documents, and a system tuned to return the single best match will answer confidently from one of them. The answer is not wrong so much as partial, which is harder to notice. If a question about leave policy needs the handbook, the local amendment and last quarter's memo, retrieving only the handbook produces an answer that is correct in isolation and wrong in effect.
What to put in the corpus, and what to keep out
The instinct is to index everything, on the grounds that more context cannot hurt. It can. Every superseded draft in the archive is a passage that can be retrieved and presented with the same authority as the current version, and the model has no way to know which is which unless you tell it.
- Prefer a smaller, curated corpus over a complete one, at least to begin with. Fifty documents someone vouches for beat five thousand nobody has looked at.
- Record dates and supersession where you can. "Effective from" in a document is information the retrieval step can use.
- Keep drafts out. If they must be in, keep them in a separate collection that is not searched by default.
- Revisit after a month. The corpus that made sense in planning is rarely the one people actually ask about.
Permissions are part of the answer
The failure mode here is subtle enough to be worth restating. If retrieval runs over everything and access control is applied to the final answer, the model has already read material the person was not cleared for, and a sufficiently indirect question can extract its substance without ever quoting it. The filter must sit in front of retrieval, not behind generation.
This has an organizational consequence people underestimate: your document permissions are about to be tested. Retrieval will find the folder that was shared with everyone in 2019 because it was easier at the time. Treat the first month as a permissions audit that happens to produce answers.
How to tell whether it is working
Collect thirty real questions from the people who will use it, with answers agreed by someone who knows. Run them. Read not just whether the answer was right, but whether the retrieved passages were the ones a knowledgeable person would have opened.
When the answer is right and the sources are wrong, the system got lucky. That is the result worth investigating, because it will not stay lucky.