Noise and bias — why a written-down rule beats an expert
One question is still open. You scored a lead in the first lesson, and you are about to score it again — the same lead, the same facts, in a different order. What happens next is the reason machine learning exists.
Every pattern so far has been a habit. The same error every time, in the same direction, for everybody.
A habit can be corrected once somebody points it out. This last one cannot, and it is the reason the rest of this course exists.
Score this lead out of ten
An inbound lead. Here is everything known about it.
- Their company uses a competitor's product today.
- No budget mentioned anywhere.
- Arrived from a comparison article, not from search.
- Head of Operations at a 400-person logistics company.
- Downloaded the pricing page twice this week.
That is everything you know about them.
How good is this lead? Score it out of ten.
Keep your answer. This one comes back later.
It was the same lead, and only the order of the facts changed
Five facts in the first lesson, the same five just now. Nothing added, nothing removed, nothing reworded. Listed in a different order.
What you answered earlier
You did not answer that one.
And the same lead, with the competitor first:
What you answered earlier
You did not answer that one.
Most people give a different number the second time. If yours matched exactly, that is worth noticing rather than assuming.
This is the error nobody can see in themselves
Every pattern in the last two lessons was in principle catchable. Somebody could tell you about anchoring and you would watch for it.
You never get a second opinion from yourself. You score the lead once, you move on, and there is nothing to compare it against. The disagreement is real and it is invisible.
A habit is the same error every time. A wobble is a different error every time. Scoring every lead two points high is a habit. Scoring the same lead 7 on Monday and 5 on Thursday is a wobble, and the two behave completely differently.
A habit can be measured and subtracted. A wobble cannot, because there is nothing steady to subtract.
Two underwriters, same training, and both 6 points too generous
500 insurance cases with a true value. Two underwriters price every one of them.
Both have exactly the same habit: they run 6 points generous. That part is built into the simulation deliberately, because it is what makes the result mean something.
They still disagree with each other by 12.6 points on the same file. Same case, same rules, same training, two different answers.
A rule with the identical 6-point habit scores 6.0
Now write the rule down. Not a better rule and not a fairer one — the same 6-point habit, applied by something that cannot have an off day.
Two people, one rule, and the only difference between them is consistency.
| Habit | Wobble | Typical error | |
|---|---|---|---|
| Underwriter one | +6 | yes | 12.2 |
| Underwriter two | +6 | yes | 12.6 |
| A written-down rule | +6 | none | 6.0 |
Underwriter one
- Habit
- +6
- Wobble
- yes
- Typical error
- 12.2
Underwriter two
- Habit
- +6
- Wobble
- yes
- Typical error
- 12.6
A written-down rule
- Habit
- +6
- Wobble
- none
- Typical error
- 6.0
The rule is exactly as biased as the two underwriters. It is 6 points generous on every single case, and nothing about it is fairer than they are.
51% of their error was wobble, and consistency alone removed it
The rule halves the error while knowing nothing the underwriters did not know. No extra data, no better judgment.
It is simply never inconsistent. That is the entire difference, and here it is worth 51% of the error.
A model is consistent. It is not unbiased. The 6 points are still there, on every case, forever. A model keeps whatever habit is in the data it learned from, and four later paths in this track are about exactly that.
This is the honest version of a claim you will hear often. Machines do not remove bias. They remove disagreement, and they hold whatever bias they were given perfectly steady.
So the question became: where does the rule come from?
If a written-down rule beats an expert, the obvious move is to write more rules. That was the field's first answer, and people spent thirty years on it.
It works until the rule has to be written for something nobody can explain. You know a good lead when you see one, and you cannot write down how.
Machine learning is what happens when the rule is derived from examples instead of written by hand. That is the story of the rest of this path.
What you can now say
- Judgment fails in patterns, not at random. Eight of them in this module, every one measured and repeated for decades.
- Three of those patterns are one shape. The data you were shown is not the data that exists. The failed founders, the bird's thousand mornings, and the flag that is mostly false.
- Half of an expert's error can be inconsistency. 51% of it here, removed by writing the rule down and changing nothing else.
- A model is consistent, not unbiased. It keeps the habit and loses the wobble. Knowing which is which is most of what the rest of this course is about.
Next: what the word AI actually means, and which of its four definitions can be scored.
content/notebooks/introduction-to-ai/where-judgment-fails.ipynb
Give the two underwriters different habits and watch what the rule can and cannot fix — it removes the wobble either way, and it never touches the habit. Also worth trying: raise the churn base rate from 5% to 30% and watch a flag become worth believing.
Show the code4 cells
PASSENGERS, RATE, ACC = 50_000_000, 1 / 10_000_000, 0.999
wanted = round(PASSENGERS * RATE)
ordinary = PASSENGERS - wanted
caught = round(wanted * ACC)
false_alarms = round(ordinary * (1 - ACC))
alarms = caught + false_alarms
record('air_passengers', f'{PASSENGERS:,}')
record('air_wanted', wanted)
record('air_accuracy', f'{ACC:.1%}')
record('air_caught', caught)
record('air_false', f'{false_alarms:,}')
record('air_alarms', f'{alarms:,}')
record('air_worth', f'1 in {round(alarms / caught):,}')
record('air_per_day', f'{round(false_alarms / 365):,}')
record('air_columns', ['', 'System raises an alarm', 'It does not', 'All'])
record('air_rows', [
['Actually wanted', f'{caught}', f'{wanted - caught}', f'{wanted}'],
['An ordinary passenger', f'{false_alarms:,}', f'{ordinary - false_alarms:,}', f'{ordinary:,}'],
['All', f'{alarms:,}', f'{PASSENGERS - alarms:,}', f'{PASSENGERS:,}'],
])
print(f'{caught} real of {alarms:,} alarms — about 1 in {round(alarms/caught):,}')
print(f'{round(false_alarms/365):,} innocent people stopped every day')SINGLE = 1 / 8_500 # roughly the rate the court was given
record('clark_single', '1 in 8,500')
record('clark_squared', f'1 in {round(1 / SINGLE**2):,}')
# If a first cot death makes a second more likely — as later evidence
# suggested — the honest figure moves by orders of magnitude.
record('clark_columns', ['If a first death makes a second…', 'Chance of two'])
record('clark_rows', [
['no more likely (what the court was told)', f'1 in {round(1 / SINGLE**2):,}'],
['5 times more likely', f'1 in {round(1 / (SINGLE * SINGLE * 5)):,}'],
['10 times more likely', f'1 in {round(1 / (SINGLE * SINGLE * 10)):,}'],
])
for mult in (1, 5, 10):
print(f'{mult:>3}x -> 1 in {round(1 / (SINGLE * SINGLE * mult)):,}')n = 200
skill = rng.normal(100, 12, n)
q1 = skill + rng.normal(0, 18, n)
q2 = skill + rng.normal(0, 18, n)
top, bot = np.argsort(q1)[-20:], np.argsort(q1)[:20]
record('rep_n', n)
record('top_q1', f'{q1[top].mean():.0f}')
record('top_q2', f'{q2[top].mean():.0f}')
record('top_got_worse', f'{(q2[top] < q1[top]).mean():.0%}')
record('bot_q1', f'{q1[bot].mean():.0f}')
record('bot_q2', f'{q2[bot].mean():.0f}')
record('bot_improved', f'{(q2[bot] > q1[bot]).mean():.0%}')
def plot_reps(ax):
for grp, colour, label in ((top, '#e2574c', 'best 10% of quarter one'),
(bot, '#3b6fd4', 'worst 10% of quarter one')):
for i in grp:
ax.plot([0, 1], [q1[i], q2[i]], color=colour, alpha=0.16, lw=1)
ax.plot([0, 1], [q1[grp].mean(), q2[grp].mean()], color=colour, lw=2.6, label=label)
ax.axhline(skill.mean(), color='#9aa0aa', ls='--', lw=1)
ax.annotate('everyone\'s average', (1, skill.mean()), textcoords='offset points',
xytext=(-6, 6), ha='right', fontsize=9, color='#77777f')
ax.set_xticks([0, 1]); ax.set_xticklabels(['Quarter one', 'Quarter two'])
ax.set_ylabel('% of target')
ax.legend(frameon=False, loc='upper center')
ax.grid(axis='x', visible=False)
save_fig('nobody-was-praised', plot_reps, figsize=(6.8, 4.0))
print(f'best 10%: {q1[top].mean():.0f} -> {q2[top].mean():.0f}, {(q2[top]<q1[top]).mean():.0%} got worse')
print(f'worst 10%: {q1[bot].mean():.0f} -> {q2[bot].mean():.0f}, {(q2[bot]>q1[bot]).mean():.0%} improved')HABIT, WOBBLE, M = 6.0, 11.0, 500
truth = rng.normal(50, 15, M)
judge1 = truth + HABIT + rng.normal(0, WOBBLE, M)
judge2 = truth + HABIT + rng.normal(0, WOBBLE, M)
rule = truth + HABIT
err = lambda p: float(np.sqrt(np.mean((p - truth) ** 2)))
e1, e2, er = err(judge1), err(judge2), err(rule)
disagree = float(np.mean(np.abs(judge1 - judge2)))
record('judge_habit', f'{HABIT:.0f}')
record('judge_cases', M)
record('judge1_error', f'{e1:.1f}')
record('judge2_error', f'{e2:.1f}')
record('rule_error', f'{er:.1f}')
record('judges_disagree_by', f'{disagree:.1f}')
record('wobble_share', f'{(e1 - er) / e1:.0%}')
record('judge_columns', ['', 'Habit', 'Wobble', 'Typical error'])
record('judge_rows', [
['Underwriter one', f'+{HABIT:.0f}', 'yes', f'{e1:.1f}'],
['Underwriter two', f'+{HABIT:.0f}', 'yes', f'{e2:.1f}'],
['A written-down rule', f'+{HABIT:.0f}', 'none', f'{er:.1f}'],
])
def plot_judges(ax):
ax.bar(['Underwriter\none', 'Underwriter\ntwo', 'A written-down\nrule'],
[e1, e2, er], color=['#9aa0aa', '#9aa0aa', '#e2574c'])
for i, v in enumerate([e1, e2, er]):
ax.text(i, v + 0.25, f'{v:.1f}', ha='center', fontsize=11)
ax.axhline(HABIT, color='#3b6fd4', ls='--', lw=1.2)
ax.annotate('the habit they all share', (2.4, HABIT), textcoords='offset points',
xytext=(0, 6), ha='right', fontsize=9, color='#3b6fd4')
ax.set_ylabel('Typical error')
ax.grid(axis='x', visible=False)
save_fig('habit-and-wobble', plot_judges, figsize=(6.6, 3.8))
print(f'judges {e1:.1f} and {e2:.1f}; rule {er:.1f}; they disagree by {disagree:.1f}')
print(f'{(e1-er)/e1:.0%} of the error was wobble')Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read