What learning is — the robot with a bow
A robot that pulls left and high, and cannot see the target. Two of them, identical except that one is told where each arrow landed. That turns out to be the whole difference between a machine that learns and one that does not.
From writing rules to learning them ended with an instruction: do not write the rules, show the machine examples and let it work them out. It never said what showing a machine examples means. That is this path's job.
Here is the picture to hold on to. Every kind of learning in this course is a version of it. A robot with a bow.
It shoots at a target it cannot see. Its arrows land where it aimed, plus two things it does not know about.
A habit is the same error every time. A wobble is different every time.
- A habit. The bow pulls left and high, about 13 centimetres off, in the same direction on every shot. Nothing random about it.
- A wobble. About 4 centimetres of scatter, different on every arrow. Wind, string, the grain of the shaft.
Learning can remove the habit. It can never remove the wobble. Keep those two apart and half the disappointment with machine learning goes away. Most of the time somebody is unhappy with a model, they are asking it to remove a wobble.
Ten arrows, all aimed at the bullseye, all landing left and high
The robot did nothing wrong. The bow did the same thing ten times. Then the wobble spread the result. Both of the things this lesson measures are already in that picture.
The robot has two settings and nothing else
Everything the robot can change is here, and there are two of them. Direction is where it points, left or right. Tension is how hard the bow is drawn, so the arrow falls short or flies high.
- Arrows
- 0
- Last arrow
- —
- Last ten, average
- —
- In the gold
- 0 / 0
Learning is those two numbers moving. Nothing else about the robot can change. Turn the coach on and watch them move on their own.
Told nothing, the robot never improves. Told where its arrow landed, it does.
Build a second robot. Same bow, same habit, same wobble, same starting aim. One thing differs, and it is not inside the robot. It is what happens after the arrow lands.
- Told nothing. It aims at the bullseye and shoots. It has no way of knowing anything is wrong.
- Told where it landed. Somebody says forty centimetres high and to the left. It shifts its aim a fraction of the way back and shoots again.
That is the only difference between the two robots. Same bow, same habit, same wobble, same starting aim.
Both robots shoot 60 arrows, 400 times over. That is the whole experiment.
Both start at 13.7 centimetres. Sixty arrows later one is at 13.5 and the other is at 5.2.
That gap is the subject of this Academy. Neither robot got a better bow, a better algorithm or a better starting aim. One of them was given feedback.
The best possible score with this bow is 5 centimetres
The dotted line on that chart is the wobble on its own. It is what an archer with no habit at all would average with this bow, and it sits at 5 centimetres.
The robot settles at 5.2 and stops improving. There is nothing left for it to learn.
A model that stops improving has not necessarily failed. It may have reached the floor of its problem. Ask what the best possible score would be before being disappointed by the actual one.
The early arrows are off to one side. The late ones are spread around the middle.
Learning removed the habit and left the wobble alone. Off to one side is a habit. Spread around the middle is a wobble. 5 of the first ten arrows landed in the gold, and 10 of the last ten did.
The first ten arrows are worth 3.2 centimetres. The last ten are worth 0.
The first ten arrows after the start buy 3.2 centimetres. The last ten buy 0.
The tenth example is worth more than the thousandth. So a small pilot tells you most of what a large one would. And the answer to a disappointing model is rarely more data.
Learning is performance at a task improving with experience of it
That is the whole definition. Every word in it can be checked: a task, a way of scoring it, and more of something happening over time.
What are we actually talking about? defined an agent that perceives and acts against a performance measure. This definition sits on top of that one, with a single thing added. The performance measure is now allowed to change the agent.
Change what the robot is told and you change what kind of learning it does
Everything above rested on one line of the experiment: somebody says forty centimetres high and to the left.
Told the exact miss, the robot is doing one thing. Told nothing at all, it can still find something out. Told only hit or miss, it can still get there slowly. Those are the next three lessons, and it is the same robot every time.
content/notebooks/what-learning-is/the-robot-with-a-bow.ipynb
Change the wobble, the habit, or how far the robot moves its aim after each arrow. Worth trying: set the correction so high that it overshoots every time.
Show the code6 cells
import random
# The robot has exactly two settings it can change, and they are its
# PARAMETERS: how it is pointed, and how hard it is drawn.
# x — direction: negative is left of the bullseye, positive is right
# y — tension: negative falls short and low, positive flies high
# Its habit is one fixed error on each: pointed left, drawn too hard.
BIAS = (-11.0, 7.0) # cm: (direction, tension), every single shot
WOBBLE = 4.0 # cm: irreducible
ARROWS = 60
TRIALS = 400
GOLD = 10.0 # cm: inside this ring counts as a hit
def shoot(aim, rng):
"""Where the arrow actually lands, given where it was aimed."""
return (aim[0] + BIAS[0] + rng.gauss(0, WOBBLE),
aim[1] + BIAS[1] + rng.gauss(0, WOBBLE))
def miss_by(shot):
return (shot[0] ** 2 + shot[1] ** 2) ** 0.5
record('bias_cm', round((BIAS[0] ** 2 + BIAS[1] ** 2) ** 0.5, 1))
record('bias_direction', BIAS[0])
record('bias_tension', BIAS[1])
record('wobble_cm', WOBBLE)
record('arrows', ARROWS)
record('trials', TRIALS)
record('gold_cm', int(GOLD))import matplotlib.pyplot as plt
import matplotlib.patches as patches
def draw_range(ax, shots=(), gold=GOLD, archer=True, colour='#ee785b', hollow=False):
"""Target, archer and arrows — the picture every number in this notebook is about."""
for r, shade in zip((30, 20, gold), ('#f1f2f4', '#e4e6ea', '#d5d8dd')):
ax.add_artist(patches.Circle((0, 0), r, facecolor=shade, edgecolor='none', zorder=0))
ax.plot([0], [0], '+', color='#3a3a3a', markersize=9, mew=1.4, zorder=1)
if archer: # standing below, seen from behind
ax.plot([0, 0], [-54, -42], color='#3a3a3a', lw=1.5) # body
ax.plot([-4, 0, 4], [-60, -54, -60], color='#3a3a3a', lw=1.5) # legs
ax.plot([-8, 0, 5], [-45, -44, -42], color='#3a3a3a', lw=1.5) # arms
ax.add_artist(patches.Circle((0, -38), 3, fc='none', ec='#3a3a3a', lw=1.5))
ax.add_artist(patches.Arc((-9, -45), 7, 15, theta1=90, theta2=270, color='#3a3a3a', lw=1.5))
if len(shots):
ax.scatter([s[0] for s in shots], [s[1] for s in shots], s=42, linewidths=1.6,
facecolors='none' if hollow else colour, edgecolors=colour, zorder=2)
ax.set_xlim(-40, 40); ax.set_ylim(-64, 40); ax.set_aspect('equal')
ax.set_xticks([]); ax.set_yticks([]); ax.grid(False)
for side in ('top', 'right', 'bottom', 'left'):
ax.spines[side].set_visible(False)
rng = random.Random(4)
first_ten = [shoot((0.0, 0.0), rng) for _ in range(10)]
save_fig('the-range', lambda ax: draw_range(ax, first_ten, hollow=True), figsize=(5.0, 5.6))STEP = 0.15
def stubborn(seed):
rng = random.Random(seed)
return [miss_by(shoot((0.0, 0.0), rng)) for _ in range(ARROWS)]
def listens(seed):
rng = random.Random(seed)
aim, misses = [0.0, 0.0], []
for _ in range(ARROWS):
shot = shoot(tuple(aim), rng)
misses.append(miss_by(shot))
# The feedback: where it landed. Move BOTH parameters a fraction of
# the way back — direction from the sideways miss, tension from the
# up-and-down one. Learning is these two numbers changing.
aim[0] -= STEP * shot[0]
aim[1] -= STEP * shot[1]
return misses
record('step', STEP)
def average_curve(robot):
runs = [robot(s) for s in range(TRIALS)]
return [sum(r[i] for r in runs) / TRIALS for i in range(ARROWS)]
curve_stubborn = average_curve(stubborn)
curve_listens = average_curve(listens)
print('first arrow :', round(curve_stubborn[0], 1), 'cm vs', round(curve_listens[0], 1), 'cm')
print('last arrow :', round(curve_stubborn[-1], 1), 'cm vs', round(curve_listens[-1], 1), 'cm')record('first_miss', round(curve_listens[0], 1))
record('last_miss', round(curve_listens[-1], 1))
record('stubborn_last', round(curve_stubborn[-1], 1))
record('floor', round(sum(curve_listens[-10:]) / 10, 1))
# The wobble alone, with no bias at all: the best any archer could ever do.
rng = random.Random(99)
perfect = sum(miss_by((rng.gauss(0, WOBBLE), rng.gauss(0, WOBBLE)))
for _ in range(20000)) / 20000
record('unbeatable', round(perfect, 1))
def plot_curves(ax):
x = range(1, ARROWS + 1)
ax.plot(x, curve_stubborn, color='#9aa1ab', linewidth=2, label='told nothing')
ax.plot(x, curve_listens, color='#ee785b', linewidth=2.4, label='told where it landed')
ax.axhline(perfect, color='#6b7280', linestyle=':', linewidth=1.2)
ax.text(ARROWS * 0.55, perfect + 0.8, 'the wobble — nothing can beat this',
fontsize=9.5, color='#6b7280')
ax.set_xlabel('arrows shot')
ax.set_ylabel('distance from the bullseye (cm)')
ax.set_ylim(0, max(curve_stubborn) * 1.15)
ax.legend(frameon=False, fontsize=10)
ax.spines[['top', 'right']].set_visible(False)
save_fig('learning-curve', plot_curves, figsize=(7.4, 4.2))rng = random.Random(7)
aim, shots = [0.0, 0.0], []
for _ in range(ARROWS):
shot = shoot(tuple(aim), rng)
shots.append(shot)
aim[0] -= STEP * shot[0]
aim[1] -= STEP * shot[1]
early, late = shots[:10], shots[-10:]
record('early_in_gold', sum(1 for s in early if miss_by(s) <= GOLD))
record('late_in_gold', sum(1 for s in late if miss_by(s) <= GOLD))
def plot_target(ax):
draw_range(ax, early, colour='#9aa1ab', hollow=True)
ax.scatter([s[0] for s in late], [s[1] for s in late], s=42, color='#ee785b', zorder=3)
ax.legend(handles=[
plt.Line2D([], [], marker='o', ls='', mfc='none', mec='#9aa1ab', label='first ten arrows'),
plt.Line2D([], [], marker='o', ls='', color='#ee785b', label='last ten arrows'),
], frameon=False, fontsize=10, loc='lower right')
save_fig('on-the-target', plot_target, figsize=(5.0, 5.6))blocks = [(i, sum(curve_listens[i:i + 10]) / 10) for i in range(0, ARROWS, 10)]
def gain(i):
g = round(blocks[i - 1][1] - blocks[i][1], 1)
return 0.0 if g == 0 else g # a block that bought nothing is 0.0, not -0.0
gains = [gain(i) for i in range(1, len(blocks))]
record('gain_first', gains[0])
record('gain_last', gains[-1])
def plot_gains(ax):
labels = [f'{b[0] + 1}–{b[0] + 10}' for b in blocks[1:]]
bars = ax.bar(labels, gains, color=['#ee785b'] + ['#9aa1ab'] * (len(gains) - 1))
for b, v in zip(bars, gains):
ax.text(b.get_x() + b.get_width() / 2, v + 0.12, f'{v}', ha='center', fontsize=10)
ax.set_ylabel('cm closer than the block before')
ax.set_xlabel('arrows')
ax.spines[['top', 'right']].set_visible(False)
save_fig('what-each-block-bought', plot_gains, figsize=(7.4, 3.8))
print('gains per block of ten:', gains)Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read