Module · What learning is

Supervised learning — being told the answer

Lesson 2 of 6 · 12 min

The robot with a bow ended on the sentence the next three lessons are built from. Change what the robot is told, and you change what kind of learning it is doing.

This is the first change, and the most generous one. Somebody stands at the target and tells the robot exactly where the arrow went.

A label is the answer, written down before the machine needed it

Call that somebody the coach. Every time they speak, the robot gets one thing.

The answer. Not a hint and not a score. The actual answer to the question the robot was trying to get right, for one arrow.
Written before the robot needed it. The coach could see the target. The robot never could. That gap is the whole arrangement.

Every labelled dataset in the world is somebody's coaching, written down. Four thousand leads marked won or lost. A folder of invoices somebody already filed. Learning from labels is learning from work a human already did.

Ten labelled arrows buy 95% of what sixty buy

The coach costs money, so ask how many labels you actually need. Label the first few arrows, send the coach home, and let the robot finish the round alone.

Where the robot ends up, against how long the coach stayed.what-learning-is/being-told-the-answer.ipynb

The line drops almost all the way over the first ten labels, then flattens. Everything after that is millimetres.

Sixty arrows either way. Only the number of labels changes.

Labelled arrows	Where the robot ends up
none	13.7 cm
five	7.4 cm
ten	5.7 cm
all sixty	5.2 cm

none

Where the robot ends up: 13.7 cm

five

Where the robot ends up: 7.4 cm

ten

Where the robot ends up: 5.7 cm

all sixty

Where the robot ends up: 5.2 cm

Ten labels bought 95% of everything the coach had to give. The remaining fifty bought the last few millimetres, and they cost five times as much as the ten that did the work.

Label a hundred before you pay for ten thousand

The shape of that curve is not a fact about archery. It is the reason for three habits.

Label a hundred first. The small batch tells you almost everything the large one would, including whether the problem can be learned at all.
A disappointing model is rarely short of data. If a hundred labels and a thousand land in the same place, the thousandth was never the problem.
Budget the labelling, not the model. The coach is the expensive part of nearly every supervised project. It is also the part that cannot be automated away.

Find out what the hundredth label changes before buying the ten-thousandth. It is the cheapest experiment in the project and almost nobody runs it.

A label can be a number or a category

Now stop changing how many labels there are and change how much each one says.

A number. Eleven left and seven high. The robot learns how far to move and in which direction. Predicting a number is called regression.
A category. Left. High. Which side, and nothing more. The robot can only take a fixed step that way. Predicting which bucket something falls in is called classification.

Both are labels and both are supervised learning. They are one idea with two scorecards. The same model will do either.

A category takes 18 arrows where a number takes 10, and both end at 5.2

Sixty arrows each, averaged over four hundred rounds.what-learning-is/being-told-the-answer.ipynb

Both finish at 5.2 centimetres, because the same habit was there to remove and the same wobble stopped them both. The number-fed robot got there in 10 arrows. The category-fed one needed 18.

A vaguer label is a slower label, and it still arrives. It buys the same destination for roughly twice the examples. Make that trade on purpose, because a coarse label is often much cheaper to collect.

The coach in your world is whoever already wrote the answer down

Every supervised problem has a coach, whether or not anybody called it that. The useful question is who wrote the answer down, and what kind of answer it is.

Five ordinary questions, and the human work each one is learning from.

The question	The label — and who wrote it	Kind
Will this lead convert?	won or lost, on past leads, by whoever closed them	category
What will this deal close at?	the final amount on every closed deal	number
Is this ticket urgent?	the priority an agent set by hand	category
How long until this ships?	the days it actually took, from the log	number
Is this invoice a duplicate?	yes or no, marked at month end by finance	category

Will this lead convert?

The label — and who wrote it: won or lost, on past leads, by whoever closed them
Kind: category

What will this deal close at?

The label — and who wrote it: the final amount on every closed deal
Kind: number

Is this ticket urgent?

The label — and who wrote it: the priority an agent set by hand
Kind: category

How long until this ships?

The label — and who wrote it: the days it actually took, from the log
Kind: number

Is this invoice a duplicate?

The label — and who wrote it: yes or no, marked at month end by finance
Kind: category

None of those labels were created for a model. They are what is left over from somebody doing their job. So the first place to look for training data is your own records of what already happened.

Supervised learning is learning from examples that already have answers

That is the definition. The two questions it always raises are the two this lesson measured: how many answers, and how much does each one say?

It is also the only kind of learning that can be marked. Because the answers exist, you can hold some back, ask the machine, and count how often it was right. What are we actually talking about? said that a performance measure somebody chose is what makes a machine judgeable at all.

All of it assumed the coach exists

Somebody could see the target, was willing to watch sixty arrows, and knew the right answer to give.

Most of the data in the world has no coach behind it. Nobody marked it, nobody has time to, and often nobody knows what the right answer would be. That is the next lesson.

Labels, how many of them, and how much each one says

content/notebooks/what-learning-is/being-told-the-answer.ipynb

Download Open in Colab

Change how many arrows the coach labels, or what the coach is allowed to say. Worth trying: give the category-fed robot twice as many arrows and see whether it catches up.

Show the code3 cells

The same bow, the same habit, the same unavoidable wobble

import random

BIAS = (-11.0, 7.0)      # cm: pulls left and high, every single shot
WOBBLE = 4.0             # cm: irreducible
ARROWS = 60
TRIALS = 400
STEP = 0.15              # how far it moves its aim when told the exact miss

def shoot(aim, rng):
    return (aim[0] + BIAS[0] + rng.gauss(0, WOBBLE),
            aim[1] + BIAS[1] + rng.gauss(0, WOBBLE))

def miss_by(shot):
    return (shot[0] ** 2 + shot[1] ** 2) ** 0.5

rng = random.Random(99)
FLOOR = sum(miss_by((rng.gauss(0, WOBBLE), rng.gauss(0, WOBBLE)))
            for _ in range(20000)) / 20000
record('floor', round(FLOOR, 1))
record('arrows', ARROWS)

Labelled arrows against where the robot ends up

BUDGETS = [0, 5, 10, 20, 40, 60]

def run(seed, labelled):
    """Shoot ARROWS arrows, hearing from the coach only for the first `labelled`."""
    rng = random.Random(seed)
    aim, misses = [0.0, 0.0], []
    for i in range(ARROWS):
        shot = shoot(tuple(aim), rng)
        misses.append(miss_by(shot))
        if i < labelled:                      # the label: where it landed
            aim[0] -= STEP * shot[0]
            aim[1] -= STEP * shot[1]
    return misses

def ends_at(labelled):
    """Average miss over the last ten arrows, across TRIALS repeats."""
    runs = [run(s, labelled) for s in range(TRIALS)]
    return sum(sum(r[-10:]) / 10 for r in runs) / TRIALS

scores = [ends_at(n) for n in BUDGETS]
for n, s in zip(BUDGETS, scores):
    print(f'{n:>3} labelled arrows -> ends at {s:.1f} cm')

record('labelled_none', round(scores[0], 1))
record('labelled_five', round(scores[1], 1))
record('labelled_ten', round(scores[2], 1))
record('labelled_all', round(scores[-1], 1))
record('share_from_ten', round(100 * (scores[0] - scores[2]) / (scores[0] - scores[-1])))

def plot_budget(ax):
    ax.plot(BUDGETS, scores, color='#ee785b', linewidth=2.4, marker='o', markersize=7)
    ax.axhline(FLOOR, color='#6b7280', linestyle=':', linewidth=1.2)
    ax.text(28, FLOOR + 0.7, 'the wobble — nothing can beat this',
            fontsize=9.5, color='#6b7280')
    for n, s in zip(BUDGETS, scores):
        ax.annotate(f'{s:.1f}', (n, s), textcoords='offset points',
                    xytext=(0, 12), ha='center', fontsize=10)
    ax.set_xlabel('arrows the coach was there for')
    ax.set_ylabel('where it ends up (cm from the bullseye)')
    ax.set_ylim(0, max(scores) * 1.25)
    ax.spines[['top', 'right']].set_visible(False)

save_fig('what-a-label-buys', plot_budget, figsize=(7.4, 4.0))

The same sixty arrows, told a number and told a category

CAT_STEP = 0.6           # cm it shifts when all it hears is 'left' or 'high'

def sign(v):
    return 1.0 if v > 0 else (-1.0 if v < 0 else 0.0)

def run_kind(seed, kind):
    rng = random.Random(seed)
    aim, misses = [0.0, 0.0], []
    for _ in range(ARROWS):
        shot = shoot(tuple(aim), rng)
        misses.append(miss_by(shot))
        if kind == 'number':
            aim[0] -= STEP * shot[0]
            aim[1] -= STEP * shot[1]
        elif kind == 'category':
            aim[0] -= CAT_STEP * sign(shot[0])
            aim[1] -= CAT_STEP * sign(shot[1])
    return misses

def average_curve(kind):
    runs = [run_kind(s, kind) for s in range(TRIALS)]
    return [sum(r[i] for r in runs) / TRIALS for i in range(ARROWS)]

curve_number = average_curve('number')
curve_category = average_curve('category')
curve_nothing = average_curve('nothing')

NEAR = FLOOR + 1.0       # 'as good as it is ever going to get', within a centimetre
def arrows_to_near(curve):
    return next(i + 1 for i, v in enumerate(curve) if v < NEAR)

record('cat_step', CAT_STEP)
record('near_floor', round(NEAR, 1))
record('arrows_number', arrows_to_near(curve_number))
record('arrows_category', arrows_to_near(curve_category))
record('ends_number', round(sum(curve_number[-10:]) / 10, 1))
record('ends_category', round(sum(curve_category[-10:]) / 10, 1))

def plot_kinds(ax):
    x = range(1, ARROWS + 1)
    ax.plot(x, curve_nothing, color='#9aa1ab', linewidth=2, label='told nothing')
    ax.plot(x, curve_category, color='#3b6fd4', linewidth=2.2,
            label='told a category - "left, high"')
    ax.plot(x, curve_number, color='#ee785b', linewidth=2.4,
            label='told a number - "11 left, 7 high"')
    ax.axhline(FLOOR, color='#6b7280', linestyle=':', linewidth=1.2)
    ax.set_xlabel('arrows shot')
    ax.set_ylabel('distance from the bullseye (cm)')
    ax.set_ylim(0, max(curve_nothing) * 1.15)
    ax.legend(frameon=False, fontsize=10)
    ax.spines[['top', 'right']].set_visible(False)

save_fig('number-or-category', plot_kinds, figsize=(7.4, 4.2))
print('to within a centimetre of the floor:',
      arrows_to_near(curve_number), 'arrows vs', arrows_to_near(curve_category))