Choosing the kind of learning — which is your problem?
Three questions, in a fixed order, sort any problem into the kind of learning its feedback will support. The cost of sorting it wrong is not a matter of opinion, because this path measured every one of them.
Five lessons, one robot, one bow. Only what somebody was willing to say after each arrow ever changed.
This lesson turns that round and points it at your own work.
Three questions, and the order matters more than the answers
Most bad projects are question two, asked before question one.
- Is the answer already written down somewhere? A rate card, a policy, a table, an exact match. If yes, build the rule and stop.
- Has somebody already recorded the answer for past cases? Won or lost, the amount, the priority they set. If yes, that is supervised learning.
- Is there no answer at all, only a consequence? Nobody can say what you should have done, but something happened afterwards that was better or worse. That is the third kind.
If none of the three apply, you have unlabelled data and a hope. That is a real place to be and lesson three is about it. Go in expecting structure rather than answers, because that is the only thing on offer.
Five questions somebody will actually ask you
| The question | What you have | So |
|---|---|---|
| Which VAT rate applies to this invoice | a published rate | no model — arithmetic |
| What will this deal close at | the final amount on every closed deal | supervised, a number |
| Is this ticket urgent | priorities agents set by hand | supervised, a category |
| Which of our customers behave alike | nothing marked at all | unsupervised — expect structure |
| Which subject line gets more replies | only whether they replied | the third kind — consequences |
Which VAT rate applies to this invoice
- What you have
- a published rate
- So
- no model — arithmetic
What will this deal close at
- What you have
- the final amount on every closed deal
- So
- supervised, a number
Is this ticket urgent
- What you have
- priorities agents set by hand
- So
- supervised, a category
Which of our customers behave alike
- What you have
- nothing marked at all
- So
- unsupervised — expect structure
Which subject line gets more replies
- What you have
- only whether they replied
- So
- the third kind — consequences
The sort never depended on how hard the question sounded. It depended on what somebody had already written down. You can answer that in an afternoon, without a data scientist.
Over-engineering costs money once. Under-engineering costs money until somebody notices.
| The mis-sort | What it cost, measured |
|---|---|
| One word, when a number was available | 517 arrows instead of 12 — 43 times the experience for the same result |
| A category, when a number was available | 21 arrows instead of 12, and roughly twice the labels |
| A model, where measuring once was exact | the arithmetic settled at 5.2 cm against the learner's 5.3 — and was finished at arrow 10 |
| A fixed rule, where the world moves | 11.6 cm from the change onwards, permanently, against the learner's 6.1 |
| More data, when the curve had flattened | the first ten labels bought 95% of everything; the other fifty bought millimetres |
One word, when a number was available
- What it cost, measured
- 517 arrows instead of 12 — 43 times the experience for the same result
A category, when a number was available
- What it cost, measured
- 21 arrows instead of 12, and roughly twice the labels
A model, where measuring once was exact
- What it cost, measured
- the arithmetic settled at 5.2 cm against the learner's 5.3 — and was finished at arrow 10
A fixed rule, where the world moves
- What it cost, measured
- 11.6 cm from the change onwards, permanently, against the learner's 6.1
More data, when the curve had flattened
- What it cost, measured
- the first ten labels bought 95% of everything; the other fifty bought millimetres
Read the second and fourth rows together. They point opposite ways. Neither over-engineering nor under-engineering is the safe default, which is why sorting is a skill rather than a preference.
Most data is partly labelled, and nobody hands you the clean case
Every case above was clean. The one everybody actually has is not. Three hundred rows somebody marked, forty thousand nobody did.
Take lesson three's ninety arrows, mark a handful by hand, and file all ninety. There are two ways to spend those labels.
- Labels only. Ignore that the unlabelled arrows exist. For any arrow, copy the kind of the nearest marked one.
- Group first, then name. Cluster all ninety without looking at a label, then use the handful only to put a name on each group.
The second sounds right, and it is the one I expected to win. It uses every arrow, where the first throws most of the data away.
It did not win
The two lines run together the whole way, and then the simple one keeps going.
| Arrows marked by hand | Labels only | Group first, then name |
|---|---|---|
| three | 71% | level with it |
| fifteen | 96% | level with it |
| all ninety | 100% | 96% |
three
- Labels only
- 71%
- Group first, then name
- level with it
fifteen
- Labels only
- 96%
- Group first, then name
- level with it
all ninety
- Labels only
- 100%
- Group first, then name
- 96%
A composite is capped by its weakest part. Naming a group perfectly does nothing about the arrows the grouping put in the wrong group. The ceiling was never the labels. It was the clustering.
What you can learn is decided by what you are told
The technique that sounds most sophisticated is not a tiebreak. That experiment was written to confirm the opposite and refused to, which is the most useful thing it could have done.
You now have that in four measured forms. And the sorting question — what has somebody already written down? — gets you to the right one before a single line is built.
Where this goes next
This path never built a model. It sorted problems and measured what each kind of feedback was worth. Choosing the wrong kind cannot be fixed by a better algorithm, and the next paths are all algorithms.
The first of them is decision trees. They take the commonest sort in that table, where somebody recorded the answer, and learn it by asking questions and splitting on the answers. It is the first model in this Academy you will build by hand.
content/notebooks/what-learning-is/which-kind-is-your-problem.ipynb
Change how many arrows get marked by hand, or how far apart the three causes sit. Worth trying: push the causes together and see whether grouping first ever becomes the better option.
Show the code3 cells
import random
WOBBLE = 4.0
EACH = 30
KINDS = [('heavy', 0.0, -12.0), ('light', 2.0, 10.0), ('warped', -13.0, -1.0)]
def sample(seed):
rng = random.Random(seed)
points, truth = [], []
for kind, (_, dx, dy) in enumerate(KINDS):
for _ in range(EACH):
points.append((dx + rng.gauss(0, WOBBLE), dy + rng.gauss(0, WOBBLE)))
truth.append(kind)
return points, truth
record('arrows_total', len(KINDS) * EACH)
record('kinds', len(KINDS))def k_means(pts, k, rounds=30):
ordered = sorted(pts)
centres = [ordered[min(len(ordered) - 1, int((i + 0.5) / k * len(ordered)))] for i in range(k)]
assign = [0] * len(pts)
for _ in range(rounds):
for i, p in enumerate(pts):
assign[i] = min(range(k),
key=lambda c: (p[0] - centres[c][0]) ** 2 + (p[1] - centres[c][1]) ** 2)
for c in range(k):
mine = [p for p, a in zip(pts, assign) if a == c]
if mine:
centres[c] = (sum(p[0] for p in mine) / len(mine),
sum(p[1] for p in mine) / len(mine))
return assign
def labels_only(pts, truth, marked):
"""Copy the kind of the nearest arrow somebody marked."""
right = 0
for i, p in enumerate(pts):
nearest = min(marked, key=lambda q: (p[0] - pts[q][0]) ** 2 + (p[1] - pts[q][1]) ** 2)
right += truth[nearest] == truth[i]
return right / len(pts)
def group_then_name(pts, truth, marked):
"""Cluster blind, then let the marked arrows vote on what each group is called."""
assign = k_means(pts, len(KINDS))
naming = {}
for g in range(len(KINDS)):
votes = [truth[q] for q in marked if assign[q] == g]
naming[g] = max(set(votes), key=votes.count) if votes else -1
return sum(1 for i in range(len(pts)) if naming[assign[i]] == truth[i]) / len(pts)
def compare(n_labels, trials=20):
a, b = [], []
for s in range(trials):
pts, truth = sample(s)
marked = random.Random(1000 + s).sample(range(len(pts)), n_labels)
a.append(labels_only(pts, truth, marked))
b.append(group_then_name(pts, truth, marked))
return 100 * sum(a) / trials, 100 * sum(b) / trials
budgets = [3, 6, 15, 30, 90]
only, grouped = zip(*(compare(n) for n in budgets))
for n, o, g in zip(budgets, only, grouped):
print(f'{n:>2} labels -> labels only {o:5.1f}% group first {g:5.1f}%')
record('labels_3_only', round(only[0]))
record('labels_6_only', round(only[1]))
record('labels_15_only', round(only[2]))
record('labels_all_only', round(only[-1]))
record('labels_all_grouped', round(grouped[-1]))
record('grouping_ceiling', round(max(grouped)))def plot_labels(ax):
ax.plot(budgets, only, color='#ee785b', linewidth=2.4, marker='o', markersize=7,
label='labels only')
ax.plot(budgets, grouped, color='#3b6fd4', linewidth=2.2, marker='s', markersize=6,
label='group first, then name')
ax.set_xlabel('arrows somebody marked by hand')
ax.set_ylabel('all ninety filed correctly (%)')
ax.set_ylim(60, 104)
ax.legend(frameon=False, fontsize=10, loc='lower right')
ax.spines[['top', 'right']].set_visible(False)
save_fig('what-a-handful-of-labels-buys', plot_labels, figsize=(7.2, 4.0))Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read