What RAG does well
Retrieval-augmented generation, or RAG, finds the passages most relevant to a question and gives them to a language model, which writes an answer based on them. It works well when the answer is spread across prose: comparing clauses across supplier contracts, summarising a customer’s dispute history, or answering a question from a long set of internal procedures.
It works poorly for questions about numbers. “How many units did we sell by region last quarter?” needs a query against a well-defined data model, not a search for similar text. Retrieval finds related words; it does not add up transactions reliably.
Three options, in order of simplicity
- A structured query or a dashboard. If the answer depends on rows, dates, prices or statuses, query the data model. Calculations stay explicit and reproducible.
- Keyword search. If people look for exact part numbers, legal names, error codes or phrases, full-text search is simpler, faster and more predictable than anything involving a language model.
- Hybrid retrieval with a language model. If the question needs a synthesis across documents, combine keyword and semantic retrieval, and have the model draft an answer that cites the passages it used, so a person can check it.
Ten questions, three methods
| Question someone asks | Best method |
|---|---|
| What was gross margin by channel last month? | Structured query |
| Which customers have not ordered in 90 days? | Structured query |
| Where is the datasheet for part 4471-B? | Keyword search |
| Which contracts mention “liquidated damages”? | Keyword search |
| What does our travel policy say about trains versus flights? | Retrieval with a language model |
| How do our payment terms differ across our five largest suppliers? | Retrieval with a language model |
The pattern is usually obvious once the questions are written down. It is rarely obvious from a vendor demo.
A quick test before you commission anything
Write down ten real questions people ask today, and next to each one, what a correct answer looks like and where it would come from. If most answers are numbers, you need a data model. If most are exact lookups, you need search. Only if most require reading and combining several documents is RAG the right core.
Many useful systems end up combining all three: structured queries for figures, keyword search for exact terms, and a language model for synthesis, each used where it is strongest.
The parts demos leave out
- Permissions. Retrieval must respect who is allowed to see what. An assistant that can quote the board minutes to everyone is a security incident, not a feature.
- Freshness. The search index needs updating when documents change, or answers will cite a policy that no longer applies.
- Evaluation. Collect real questions with known good answers and test against them on every change, so quality is measured rather than assumed.
- Cost per answer. Language models are paid per use. A design that sends whole documents to the model for every question can become expensive quickly.