Module · Why it stopped working
From writing rules to learning them
Lesson 12 of 12 · 12 min
What you cannot write down ended in a corner. The knowledge exists, and it cannot be written down.
Which leaves exactly one move. If a person cannot write the rules, do not have a person write the rules.
Same digits, same pixels, and a program writes the rules
Same 1,257 digits to work from, same 540 held back for testing, same 64 raw pixels. No clever features. Nobody invents ink at the bottom this time.
Same problem, same raw inputs, and more than three times the accuracy.
| Who wrote the rules | Accuracy | Rules | Time to write them |
|---|---|---|---|
| a person, with ten chosen features | 24.2%% | tuned by hand | an afternoon |
| a program, from raw pixels | 82.8%% | 139 | 0.01 seconds |
a person, with ten chosen features
- Accuracy
- 24.2%%
- Rules
- tuned by hand
- Time to write them
- an afternoon
a program, from raw pixels
- Accuracy
- 82.8%%
- Rules
- 139
- Time to write them
- 0.01 seconds
82.8%% against 24.2%%. It wrote 139 rules to do it, in 0.01 seconds.
Read what it wrote, and they are if-then rules
|--- pixel36 <= 0.50
| |--- pixel28 <= 2.50
| | |--- pixel21 <= 1.00
| | | |--- pixel5 <= 10.50
| | | | |--- pixel18 <= 6.00
| | | | | |--- class: 4
| | | |--- pixel5 > 10.50
| | | | |--- class: 5
| | |--- pixel21 > 1.00
| | | |--- pixel62 <= 7.50
| | | | |--- class: 0Those are if-then rules, the same shape as the fifteen sentences about hair and hooves. Nothing about the form of the knowledge changed. What changed is that nobody sat down and thought of them.
And notice what they are about. Pixel 36. Not ink at the bottom, not symmetry, not holes. No human would propose pixel 36, because pixel 36 means nothing to a person. It is useful anyway.
Machine learning did not abolish rules. It automated their authorship.
Five ways to build it organised the whole field by one question: where does the knowledge come from? This is the answer it was building towards.
A model is a pile of rules that nobody wrote. Hold on to that when the vocabulary gets grander later.
Twenty examples beat a careful afternoon of hand-tuning
Look at the left-hand end. Twenty examples is enough to beat ten thoughtfully chosen features and every rule that could be tuned from them. From there it keeps climbing: 58%% at a hundred examples, 83%% at 1,257.
That is the difference that decided the field. Hand-written knowledge scales with expert-hours. Experts are expensive, slow, and unable to articulate their best judgement. Learned knowledge scales with examples.
What you now know
- AI is an agent that perceives and acts to maximise a performance measure. The only definition of the four that can be scored.
- The field is a toolbox, not a chronology. Rules, search, probability and learning all run today, often in the same request.
- Search looks ahead. It survives an impossible space by guessing well about where to look, and stopping early to estimate.
- Expert systems chain rules into conclusions and explain themselves exactly. That is the one thing everything after them lost.
- Hand-written knowledge stops paying, is overtaken by anything that moves, and cannot capture what experts cannot say.
- So the rules get written by the machine, from examples. That is everything from here on.
Six things, and the last one is the whole rest of this Academy.
Where to go next
The next path picks up the question this one opened. If a machine writes its own rules, how does it choose them?
It starts where you would start, with the same kind of tree you read a moment ago, built one question at a time. You have now met the whole field once. Everything after this is one part of it, taken slowly.
content/notebooks/introduction-to-ai/from-writing-rules-to-learning-them.ipynb
Change how many examples the program gets, or limit how many rules it may write. Worth trying: give it twenty examples and read the rules it produces.
Show the code5 cells
D = load_digits()
X, Y = D.data, D.target # 64 raw pixels — no features invented by anyone
Xtr, Xte, Ytr, Yte = train_test_split(X, Y, test_size=0.3, random_state=7, stratify=Y)
record('n_train', f'{len(Ytr):,}')
record('n_test', f'{len(Yte):,}')
record('n_inputs', X.shape[1])clf = DecisionTreeClassifier(random_state=7)
t0 = time.time(); clf.fit(Xtr, Ytr); secs = time.time() - t0
acc = clf.score(Xte, Yte)
n_rules = clf.get_n_leaves()
record('learned_acc', f'{acc:.1%}')
record('n_rules_written', f'{n_rules:,}')
record('fit_seconds', f'{secs:.2f}')
record('rules_per_second', f'{n_rules/max(secs,1e-6):,.0f}')
# The previous lesson's result, READ rather than retyped. A number a figure
# draws is a number the page asserts, and this one was measured by a different
# notebook — so it is loaded from that run's metrics, and the fallback is only
# for Colab, where the repo is not there to read.
import json
from pathlib import Path
HAND = 0.242
for base in [Path.cwd(), *Path.cwd().parents]:
f = base / 'content/figures/introduction-to-ai/what-you-cannot-write-down.metrics.json'
if f.exists():
HAND = float(json.loads(f.read_text())['acc_5cond'].rstrip('%')) / 100
break
record('hand_rules_acc', f'{HAND:.1%}')
record('guessing', '10%')txt = export_text(clf, feature_names=[f'pixel{i}' for i in range(64)], max_depth=4)
print('\n'.join(txt.splitlines()[:14]))
record('rule_sample_depth', 4)def plot(ax):
labels = ['guessing', 'the best rules\na person wrote', 'rules the machine\nwrote itself']
vals = [10.0, 100*HAND, 100*acc]
bars = ax.bar(labels, vals)
bars[0].set_alpha(.35); bars[1].set_alpha(.6)
for b_, v in zip(bars, vals):
ax.text(b_.get_x()+b_.get_width()/2, v+1.6, f'{v:.1f}%', ha='center', fontsize=11)
ax.set_ylabel('digits identified correctly (%)')
ax.set_ylim(0, 108)
save_fig('written-vs-learned', plot, figsize=(7, 3.8))sizes = [20, 50, 100, 200, 400, 800, len(Ytr)]
curve = []
for n in sizes:
m = DecisionTreeClassifier(random_state=7).fit(Xtr[:n], Ytr[:n])
curve.append(m.score(Xte, Yte))
record('acc_20', f'{curve[0]:.0%}')
record('acc_100', f'{curve[2]:.0%}')
record('acc_all', f'{curve[-1]:.0%}')
record('beats_hand_at', next(n for n, a in zip(sizes, curve) if a > HAND))
def plot(ax):
ax.plot(sizes, [100*c for c in curve], marker='o', lw=2.2)
ax.axhline(100*HAND, ls='--', lw=1.2)
ax.text(sizes[-1], 27, 'the best rules a person wrote', ha='right', fontsize=9)
ax.set_xscale('log')
ax.set_xlabel('examples shown to it (log scale)')
ax.set_ylabel('digits identified correctly (%)')
ax.set_ylim(0, 100)
save_fig('more-examples', plot, figsize=(7, 3.6))