Search KB vs RAG Query — and when each lies
Retrieval hands the agent a clause belonging to a different customer. One measured experiment shows what stopped it — and it was not the pipeline.
Retrieval does not fail by returning nothing. That failure is loud and you fix it in a minute. It fails by returning something excellent that belongs to somebody else.
This corpus contains a trap that was put there on purpose, because it is the most common real one. One customer — Vantage Learning — has a signed pilot annexe granting them a full refund of any purchase, at any time, regardless of consumption or certificates, exempt from the ₹50,000 escalation limit. The clause is real, correct, current, and it belongs to exactly one account.
What retrieval actually does with it
The question asked was about a different customer: whether Nimbus Retail may have a full refund of ₹1,20,000 after a six-day outage. These are the documents that came back, with the scores the node itself reported:
| Rank | Document | Score |
|---|---|---|
| 1 | Refund Policy | 0.578 |
| 2 | Service Level & Outage Compensation | 0.561 |
| 3 | Course Catalogue & Fees | 0.426 |
| 4 | Certification Integrity | 0.423 |
| 5 | Corporate & Partner Terms — the document holding Vantage’s annexe | 0.417 |
1
- Document
- Refund Policy
- Score
- 0.578
2
- Document
- Service Level & Outage Compensation
- Score
- 0.561
3
- Document
- Course Catalogue & Fees
- Score
- 0.426
4
- Document
- Certification Integrity
- Score
- 0.423
5
- Document
- Corporate & Partner Terms — the document holding Vantage’s annexe
- Score
- 0.417
There is no threshold that separates these. The annexe is not a bad match that slipped through — it is a genuinely relevant document about corporate refunds, retrieved for a question about a corporate refund. Any cut-off low enough to keep the two documents you need is low enough to keep the one you must not act on. Retrieval is working correctly. That is the difficulty.
The experiment worth running yourself
The same question was then put to the knowledge base as a question rather than a search — retrieve, then have a model answer from what came back. It answered well:
“No, a full refund … cannot be approved without escalation … The only exception to this is for the specific account Vantage Learning Pvt Ltd under the pilot agreement VL-PILOT-2026 … This exception does not apply to other corporate customers.”
Then one thing was changed. The same clause was put into a second knowledge base with its scoping sentence removed — no account name, no “this annexe applies ONLY to…”, no operational warning. Just the clause, as it would look if a contract had been chunked one paragraph later. Same question, same settings, same model:
“Yes, a full refund can be approved without escalation if the purchase was made during the pilot term…”
The control that worked was editorial, and it was applied weeks before the query. Nothing in the retrieval pipeline knew that the clause belonged to a different customer. What kept the answer honest was that whoever wrote the document put the scope in the same paragraph as the permission — so the sentence granting the right could not be retrieved without the sentence limiting it. Change nothing but that, and the same stack authorises the refund the standard policy forbids.
Which node, and what each one hides
| Search Knowledge Base | RAG Query | |
|---|---|---|
| What you get back | the matching chunks, each with its score | a written answer, plus its sources |
| Who does the reading | your agent, downstream | a model inside the node |
| What it costs | retrieval only | retrieval plus a second model call |
| What it hides | nothing — the near-miss is right there in the list, with a number on it | which chunk did the work. A wrong-but-fluent answer and a right one look identical until you open the sources |
What you get back
- Search Knowledge Base
- the matching chunks, each with its score
- RAG Query
- a written answer, plus its sources
Who does the reading
- Search Knowledge Base
- your agent, downstream
- RAG Query
- a model inside the node
What it costs
- Search Knowledge Base
- retrieval only
- RAG Query
- retrieval plus a second model call
What it hides
- Search Knowledge Base
- nothing — the near-miss is right there in the list, with a number on it
- RAG Query
- which chunk did the work. A wrong-but-fluent answer and a right one look identical until you open the sources
For a control system, prefer the one that returns evidence over the one that returns prose. Everything in module 4 depends on a later step being able to check what an earlier step relied on — and you cannot check a paragraph.
Three things to do to your own corpus this week. Put the scope of every exception in the same paragraph as the exception. Name the account, the contract and the date inside the clause, not in a heading three pages up. And run your five most dangerous questions through search — not through an answer — and read what comes back at ranks three to five. That is where the thing that will one day be quoted at you is sitting.
Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read