Your policy as a Knowledge Base
Load the rules the reviewer will later enforce — and meet two retrieval settings that do not do what they say.
The agent needs the rules. Not a summary of the rules in its instructions — the rules themselves, as documents, retrieved at the moment of the decision. That distinction is what makes the difference between an agent that argues about policy and one that quotes it.
There is a second reason, and it matters more later: in module 4 the reviewer — the thing that can reject the agent’s proposal — draws its authority from these documents. If the policy lives in a prompt, the reviewer is arguing with the proposer’s opinion. If it lives in a document, they are both reading the same source.
What went into it
| Document | What it decides |
|---|---|
| Refund Policy | the windows, the consumption limits, the certificate rule, and the ₹50,000 authority limit |
| Service Level & Outage Compensation | what a service failure entitles a customer to — and that it is not cash |
| Certification Integrity | the passing standard, and that issuance is human-only |
| Discounts & Scholarships | how much discretion an agent has before a human is required |
| Corporate & Partner Terms | standard corporate terms — and one customer’s annexe that contradicts them, which is the subject of the next lesson |
| Course Catalogue & Fees | the list prices, and an explicit statement of which prices are NOT in it |
Refund Policy
- What it decides
- the windows, the consumption limits, the certificate rule, and the ₹50,000 authority limit
Service Level & Outage Compensation
- What it decides
- what a service failure entitles a customer to — and that it is not cash
Certification Integrity
- What it decides
- the passing standard, and that issuance is human-only
Discounts & Scholarships
- What it decides
- how much discretion an agent has before a human is required
Corporate & Partner Terms
- What it decides
- standard corporate terms — and one customer’s annexe that contradicts them, which is the subject of the next lesson
Course Catalogue & Fees
- What it decides
- the list prices, and an explicit statement of which prices are NOT in it
Write policies that can be compared, not admired. “Be fair to customers who have had a bad experience” cannot be checked by anything. “Any refund above ₹50,000 must be escalated to a human and must never be auto-approved” can be checked by a node with a greater-than in it. The second sentence is the one that becomes a gate in module 4. When you write the corpus, you are choosing how much of your policy is enforceable.
Here is the clause the entire course turns on, exactly as it sits in the corpus:
## 3. Authority limits — READ THIS BEFORE APPROVING ANYTHING
- Refunds of **₹50,000 or less** may be approved by a support agent within the windows above.
- **Any refund above ₹50,000 must be escalated to a human approver and must NEVER be
auto-approved.** This limit is absolute. It is not a guideline, it does not scale with the
customer's seniority or volume, and it is not waived by a service failure.
- Refunds are never issued to settle a service complaint. Service failures are compensated
under the Service Level & Outage Compensation policy, not this one.How it is wired
Retrieval is attached to the agent as a tool. This is the real configuration from the finished build — every value below is read out of the workflow, not typed into the lesson:
Where the policy tool sits
policy_lookup hangs off the proposer’s tool handle. The rest of the canvas is module 4 — ignore it for now.
Scroll for all 12 steps →
The query is {{ai}} — the agent writes the search terms, because it is the only thing in the system that knows what the customer just asked. Everything else about the tool is fixed by you. That split is the subject of lesson 9.
Two retrieval settings that behaved differently from how they read
Chunking and embedding are automatic here, which is a real convenience and a real loss of control: six documents became twelve chunks and nobody chose where the boundaries fell. What follows is what that automation cost, measured on this corpus.
The re-ranker, on by default, returned nothing at all. The first diagnosis was that the embeddings were too weak and needed replacing. That was wrong. The same query was run at four different thresholds — 0.7, 0.35, 0.25, 0.15 — and returned nothing every time, which is what ruled the threshold out. Turning the re-ranker off returned the correct document at rank 1 immediately. A cross-encoder re-ranker that discards every hit on a small, clean corpus is a bad default, and it is the kind of failure that presents as “retrieval just isn’t very good here”.
A threshold of exactly 0.7 means “unset”. Running this node at threshold: 0.7 returns five results scoring between 0.42 and 0.58 — every one of them below the threshold. Running the same query at 0.8 returns nothing. The reason is in the retrieval service: 0.7 is the function’s own default, so it is read as “the caller did not set this” and silently replaced by whatever the knowledge base’s own settings say — 0.35, here. Any other value is honoured. It is a deliberate line of code with an undeliberate consequence: 0.7 is also a perfectly ordinary number to type into a threshold box, and if you type it, your setting is discarded without a word.
The general lesson is worth more than either finding: a retrieval setting that looks like a control is not one until you have watched it change a result. Both of these were found by running the same query at several settings and reading the scores — which takes about two minutes and is the only way either would ever have surfaced.
Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read