12 min

Rules against models — not everything is a learning problem

The habit is the same push on every shot. So measure it once, subtract it, and stop. Three lines of arithmetic beat the learner — right up until the wind changes.

Three lessons have changed what the robot is told. This one changes something else, and asks a question The robot with a bow never asked. Why is the robot iterating at all?

The habit is fixed. It is the same push on every shot, forever. So measure it once, subtract it, and stop.

Three lines of arithmetic instead of a learner

the whole method

# Ten arrows. Then never again.
habit = average(where_the_first_ten_landed)
aim   = -habit

No loop, no learning rate, no retraining.

There is no training in that, in any useful sense of the word. It is a measurement and a subtraction. It runs in the time it takes to add ten numbers up.

The rule reaches 5.2 cm. The learner reaches 5.3.

The learner against three lines of arithmetic, on a habit that never changes.what-learning-is/not-everything-is-a-learning-problem.ipynb

The rule drops straight to its answer at arrow ten and stays there. The learner is still working its way down.

Same bow, same habit, same wobble.

	Where it settles
the rule, after 10 measuring arrows	5.2 cm
the learner, given all sixty	5.3 cm
the floor	5 cm

the rule, after 10 measuring arrows

Where it settles: 5.2 cm

the learner, given all sixty

Where it settles: 5.3 cm

the floor

Where it settles: 5 cm

The arithmetic won, and it was finished at arrow ten. On a problem that holds still, the simplest thing that could work usually does.

A model costs four things a rule does not

It is approximately right. A rule computes the answer. A model estimates it, and estimating something you could compute is a strange purchase.
It cannot be read. Three lines can be checked by anybody in the room. A fitted model has to be trusted, or investigated by somebody qualified.
It needs examples, and somebody to produce them. The rule needed ten arrows. Nothing had to be labelled, stored, versioned or refreshed.
It goes stale quietly. Anything fitted to last year keeps answering confidently this year.

A rule is a line somebody can delete. A model is a commitment to keep feeding, checking and re-fitting something. That difference outlives whichever of them scored better in the first week.

A crosswind arrives at arrow 30 and the rule never notices

Same experiment, one change. A crosswind starts at arrow 30 and does not leave. Nobody announces it.

The same two, after the wind changes.what-learning-is/not-everything-is-a-learning-problem.ipynb

Both jump when the wind arrives. Only one of them comes back down.

Neither one was told the wind had arrived.

From the change onwards	Average distance
the rule	11.6 cm, ending at 11.9
the learner	6.1 cm, back near the floor in 9 arrows

the rule

Average distance: 11.6 cm, ending at 11.9

the learner

Average distance: 6.1 cm, back near the floor in 9 arrows

The rule is wrong, permanently, and it will never notice. This is the half left out of every “just write a rule” argument. The case for learning is that it does not need somebody to spot the change.

The test is whether the answer is already written down

The question is never whether a problem sounds impressive.

Four tasks nobody should train anything for, and one nobody can avoid training for.

The task	What decides the answer	So
Tax on an invoice	a published rate	arithmetic — a model could only add error
Which discount tier a deal falls in	a policy somebody wrote	a rule, changed when the policy is
Are these two rows the same invoice	an exact match on number and amount	a lookup
Which rep owns this postcode	a table you already have	a lookup
Will this lead convert	nothing written down anywhere	a model — there is no rule to write

Tax on an invoice

What decides the answer: a published rate
So: arithmetic — a model could only add error

Which discount tier a deal falls in

What decides the answer: a policy somebody wrote
So: a rule, changed when the policy is

Are these two rows the same invoice

What decides the answer: an exact match on number and amount
So: a lookup

Which rep owns this postcode

What decides the answer: a table you already have
So: a lookup

Will this lead convert

What decides the answer: nothing written down anywhere
So: a model — there is no rule to write

Only the last row has no rule to write. Suppose somebody could tell you the answer by reading a policy, a rate card or a table. Then a model is a slower, vaguer copy of a document you already have.

Averaging ten misses is already a model

One parameter, fitted once. So the question was never model or no model. It is how much machinery this problem needs, and the answer here was far less than lesson 1 implied.

Which is why From writing rules to learning them matters more than it looked. It never said rules were obsolete. It said there are problems whose rules nobody can write down, and those are rarer than the enthusiasm suggests.

Ask what the simplest thing that could work is, and how long it would keep working

Two questions, in that order. The first stops a project that should have been a spreadsheet. The second stops a spreadsheet that should have been a project.

A rule, a learner, and a world that moves

content/notebooks/what-learning-is/not-everything-is-a-learning-problem.ipynb

Download Open in Colab

Change when the wind arrives, or how strong it is. Worth trying: make the habit drift slowly from the first arrow instead of jumping once, and see which of the two notices.

Show the code3 cells

One bow. One robot that keeps adjusting, one that measures once and stops.

import random

BIAS = (-11.0, 7.0)
WOBBLE = 4.0
ARROWS = 60
TRIALS = 400
STEP = 0.15
MEASURE = 10            # arrows the rule spends working out the habit

def miss_by(shot):
    return (shot[0] ** 2 + shot[1] ** 2) ** 0.5

def learner(seed, bias_at):
    rng = random.Random(seed)
    aim, out = [0.0, 0.0], []
    for i in range(ARROWS):
        bx, by = bias_at(i)
        shot = (aim[0] + bx + rng.gauss(0, WOBBLE), aim[1] + by + rng.gauss(0, WOBBLE))
        out.append(miss_by(shot))
        aim[0] -= STEP * shot[0]
        aim[1] -= STEP * shot[1]
    return out

def rule(seed, bias_at):
    rng = random.Random(seed)
    aim, out, seen = [0.0, 0.0], [], []
    for i in range(ARROWS):
        bx, by = bias_at(i)
        shot = (aim[0] + bx + rng.gauss(0, WOBBLE), aim[1] + by + rng.gauss(0, WOBBLE))
        out.append(miss_by(shot))
        if i < MEASURE:
            seen.append(shot)
            if i == MEASURE - 1:   # measured. subtract it. never adjust again.
                aim = [-sum(s[0] for s in seen) / MEASURE, -sum(s[1] for s in seen) / MEASURE]
    return out

def average(robot, bias_at):
    runs = [robot(s, bias_at) for s in range(TRIALS)]
    return [sum(r[i] for r in runs) / TRIALS for i in range(ARROWS)]

record('measure_arrows', MEASURE)
record('arrows', ARROWS)

The learner against three lines of arithmetic

still = lambda i: BIAS

learn_still = average(learner, still)
rule_still = average(rule, still)

rng = random.Random(99)
FLOOR = sum(miss_by((rng.gauss(0, WOBBLE), rng.gauss(0, WOBBLE))) for _ in range(20000)) / 20000

def settled(curve, frm=MEASURE + 1):
    return sum(curve[frm:]) / len(curve[frm:])

record('floor', round(FLOOR, 1))
record('rule_settled', round(settled(rule_still), 1))
record('learner_settled', round(settled(learn_still), 1))
record('learner_at_measure', round(learn_still[MEASURE], 1))
record('rule_at_measure', round(rule_still[MEASURE], 1))

print(f'after the first {MEASURE} arrows — rule {settled(rule_still):.1f} cm, '
      f'learner {settled(learn_still):.1f} cm, floor {FLOOR:.1f} cm')

def plot_still(ax):
    x = range(1, ARROWS + 1)
    ax.plot(x, learn_still, color='#9aa1ab', linewidth=2.2, label='the learner')
    ax.plot(x, rule_still, color='#ee785b', linewidth=2.4, label='measure once, subtract')
    ax.axhline(FLOOR, color='#6b7280', linestyle=':', linewidth=1.2)
    ax.axvline(MEASURE, color='#b9b2aa', linewidth=1, linestyle='--')
    ax.text(MEASURE + 1, max(learn_still) * 0.92, 'habit measured', fontsize=9.5, color='#6b7280')
    ax.set_xlabel('arrows shot')
    ax.set_ylabel('distance from the bullseye (cm)')
    ax.set_ylim(0, max(learn_still) * 1.12)
    ax.legend(frameon=False, fontsize=10)
    ax.spines[['top', 'right']].set_visible(False)

save_fig('rule-against-learner', plot_still, figsize=(7.4, 4.0))

The same two, after the wind changes at arrow thirty

SHIFT_AT = 30
WIND = (9.0, -6.0)

def windy(i):
    return BIAS if i < SHIFT_AT else (BIAS[0] + WIND[0], BIAS[1] + WIND[1])

learn_wind = average(learner, windy)
rule_wind = average(rule, windy)

after = lambda c: sum(c[SHIFT_AT:]) / len(c[SHIFT_AT:])
record('shift_at', SHIFT_AT)
record('rule_after_shift', round(after(rule_wind), 1))
record('learner_after_shift', round(after(learn_wind), 1))
record('rule_end', round(rule_wind[-1], 1))
record('learner_end', round(learn_wind[-1], 1))

# How long the learner takes to get back to within a centimetre of the floor.
back = next((i - SHIFT_AT + 1 for i in range(SHIFT_AT, ARROWS) if learn_wind[i] < FLOOR + 1), None)
record('learner_recovers_in', back)

print(f'from arrow {SHIFT_AT} on — rule {after(rule_wind):.1f} cm, learner {after(learn_wind):.1f} cm')
print(f'learner is back within a centimetre of the floor after {back} arrows')

def plot_wind(ax):
    x = range(1, ARROWS + 1)
    ax.plot(x, learn_wind, color='#2f9e6e', linewidth=2.4, label='the learner')
    ax.plot(x, rule_wind, color='#ee785b', linewidth=2.4, label='measure once, subtract')
    ax.axhline(FLOOR, color='#6b7280', linestyle=':', linewidth=1.2)
    ax.axvline(SHIFT_AT, color='#b9b2aa', linewidth=1, linestyle='--')
    ax.text(SHIFT_AT + 1, max(rule_wind) * 0.9, 'the wind changes', fontsize=9.5, color='#6b7280')
    ax.set_xlabel('arrows shot')
    ax.set_ylabel('distance from the bullseye (cm)')
    ax.set_ylim(0, max(rule_wind) * 1.12)
    ax.legend(frameon=False, fontsize=10)
    ax.spines[['top', 'right']].set_visible(False)

save_fig('when-the-world-moves', plot_wind, figsize=(7.4, 4.0))

Try this in Expedify →

Related lessons

12 min read

Base rates — what a piece of evidence is actually worth

A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.

Read

10 min read

Confirmation and survivorship — what you never looked for

Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.

Read

14 min read

Loss aversion, sunk cost and regression — what it costs you

Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.

Read