Module · Where judgment fails
Confirmation and survivorship — what you never looked for
Lesson 2 of 12 · 10 min
The last lesson was about what evidence is worth. This one is about the evidence you never asked for.
I am thinking of a rule. It generates 2, 4, 6.
I have a rule in mind that some sets of three numbers obey. 2, 4, 6 obeys it. Your job is to work out what the rule is.
You may test one triple. I will tell you whether it obeys the rule.
Which triple do you test?
Almost everybody guesses the rule is even numbers going up by two, and then tests 8, 10, 12. It obeys, which feels like progress and is not.
| The triple you test | If it obeys | If it does not |
|---|---|---|
| 8, 10, 12 | you learn nothing new | cannot happen |
| 20, 22, 24 | you learn nothing new | cannot happen |
| 3, 5, 7 | your guess was too narrow | your guess survives a real test |
8, 10, 12
- If it obeys
- you learn nothing new
- If it does not
- cannot happen
20, 22, 24
- If it obeys
- you learn nothing new
- If it does not
- cannot happen
3, 5, 7
- If it obeys
- your guess was too narrow
- If it does not
- your guess survives a real test
A test that can only agree with you is not evidence. It is a repetition. Wason ran this in 1960 and most people never propose a triple that could break their own guess.
This is the shape of every model evaluation in this course. Scoring a model on the data it was built from is 8, 10, 12. It cannot come back against you, so it tells you nothing.
Your nephew wants to drop out
He has a business idea. He points out that four of the five founders he admires most left university to build something.
Is that a good argument?
Nobody counts the people who dropped out and did not become founders. There is no list of them and no article about them, so they are missing from every version of this argument anybody has ever heard.
Dobelli opens The Art of Thinking Clearly with this. It is the same reasoning that saved bomber crews in 1943. Abraham Wald pointed out that the returning planes showed where a plane can be hit and still come back. The armour belonged where there were no holes.
Ask what would be missing from this data if the story were false. If the answer is nothing would look different, the data has taught you nothing.
Both questions had the same hole in them
- The triple that could have broken your guess. You tested one that agreed with you instead, because agreement feels like progress.
- The founders who dropped out and vanished. You did not count them, because nobody wrote them down.
Evidence that could prove you wrong is the only kind that can prove you right. In both questions it existed and neither of them went looking for it.
This is the second of the two ideas this module is really about, and the pair of them carry the rest of the course. Evidence is worth what the base rate makes it worth, and it counts only if it could have come back against you.