What a knowledge base is, and how search finds things
Four real questions, four correct answers, and scores ranging from 0.06 to 0.9993 — which is the most useful thing to know about knowledge-base search before you configure one.
Three workflows you have already read hand a question to a knowledge base and answer from what comes back. This is the module about what happens in between — and it matters because the single commonest way to break a client's AI is to configure this part sensibly.
Three knowledge bases, and who is allowed to read them
| Knowledge base | Documents | Read by |
|---|---|---|
| customer knowledge base | 19 | 3 workflow(s) |
| sales playbook (internal) | 6 | 2 workflow(s) |
| operations playbook (internal) | 6 | nobody |
customer knowledge base
- Documents
19- Read by
- 3 workflow(s)
sales playbook (internal)
- Documents
6- Read by
- 2 workflow(s)
operations playbook (internal)
- Documents
6- Read by
- nobody
The split is a permission, not a filing system. The customer-facing one holds what a stranger may be told. The internal ones hold how advisors qualify and how branches run. The chat agents are wired to the first and physically cannot read the other two — which is a far better answer to “how do we stop it saying the wrong thing” than any instruction in a prompt.
One of the three is read by no workflow at all: operations playbook (internal). It is not dead — people read it, and the rules inside it are why the automations behave as they do. But it is worth knowing which of a client's knowledge bases are load-bearing for software and which are for humans, because only one kind breaks something when it is edited.
A document is not what gets searched
Each document is split into chunks, and chunks are what a search compares against and returns. That is why a long document covering six subjects performs badly: the six get mixed into the same few chunks and none of them is strongly about anything.
It is also why Writing a document that comes back is the longest lesson in this module. How you write decides how you retrieve, and no setting compensates for it.
Four questions, asked for real
These were put to the customer-facing knowledge base — the same one the website agent uses — and this is what came back, top hit first:
| Question | Top document | Score |
|---|---|---|
| what documents do I need for a home loan? | Home loan — documents you will need | 0.9993 |
| how long does it take? | How long does a loan take to be approved? | 0.8452 |
| will applying hurt my credit score? | Will applying for a loan hurt my credit score? | 0.0613 |
| what do you charge? | What a loan costs you — fees and charges | 0.4309 |
what documents do I need for a home loan?
- Top document
- Home loan — documents you will need
- Score
0.9993
how long does it take?
- Top document
- How long does a loan take to be approved?
- Score
0.8452
will applying hurt my credit score?
- Top document
- Will applying for a loan hurt my credit score?
- Score
0.0613
what do you charge?
- Top document
- What a loan costs you — fees and charges
- Score
0.4309
Now look at that last column again
Every answer is correct, and the scores run from 0.0613 to 0.9993. A question whose answer is a document with almost exactly that title as its heading scores at the bottom of that range. A question about home-loan documents scores at the top. Both got the right document first.
A score is a ranking device, not a confidence. It tells you which chunk beat the others for this question. It does not tell you whether the answer is any good, and it is not comparable between one question and the next.
The reason is worth one sentence: when re-ranking is on — and it is on for all three of these — the number you see comes from a model that scores a chunk against this query, not from a similarity between two vectors. Those numbers are routinely tiny for a perfectly correct answer.
Which is why every threshold here is zero
A knowledge base can be told to discard results below a score. Every one of these is set to discard nothing, and now you can see why: a threshold that sounds cautious — 0.7, say — would have thrown away three of the four correct answers above and left the agent saying the handbook does not cover it.
This is the mistake to not make in a client's org. It looks like tuning, it is defensible in a meeting, and it produces an assistant that refuses to answer questions it has the answer to. If retrieval seems to be returning nothing, the threshold is the first thing to look at and almost always the cause.
Change one thing — ask it something yourself
- Open the customer-facing knowledge base and search it directly, in the words a customer would use rather than the words the documents use.
- Read the ranking, not the numbers. Is the right document first?
- Now ask something the knowledge base genuinely does not cover, and look at what comes back — it will still return its best guesses, ranked. Nothing tells you it does not know.
- That last point is the agent's job, not the search's, and it is Giving an agent a knowledge base.
Try it
- Ask the same question twice, once as a customer would and once as an advisor would, and compare the rankings.
- Predict which of the nineteen documents would come back for “can my wife apply with me?”, then check.
Next: Writing a document that comes back — a document that was in this knowledge base, was exactly on the subject, and still did not come back.
Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read