Module · Why it stopped working
The limits of hand-written rules — the arms race you lose
Lesson 10 of 12 · 13 min
The machine that reasoned ended on XCON, which failed by becoming impossible to maintain. That sounds like an excuse.
It is a real limit, and you can measure it. Here it is on something smaller than a computer factory: a spam filter.
The twentieth rule is worth less than nothing
8,000 messages, 40%% of them spam. The rules are the simplest kind: if this word appears, call it spam. Each new rule is the single best one available at that moment.
The score to beat is doing nothing. Calling every message real is right 60%% of the time, because most mail is real.
The first rule is worth 5.5 percentage points. By the fifth the filter is at 81.8%%.
| Rule number | What it added |
|---|---|
| 1st | 5.5 points |
| 20th | -0.24 points |
| 40th | -0.31 points |
1st
- What it added
- 5.5 points
20th
- What it added
- -0.24 points
40th
- What it added
- -0.31 points
Read the sign on those last two numbers. They are negative. Past a point, the rules take value away.
The filter peaks at 18 rules and nobody inside can see it
This filter is at its best after 18 rules, where it gets 94.2%% of the mail right. Keep going to forty and it falls to 89.8%%.
And there is no adversary in that experiment. Nothing is fighting back. The mail is exactly the same on rule forty as it was on rule one.
Every one of those rules looked good on the mail on your desk. Rule 25 was the best rule available for the mail you had. It was not a fact about spam.
Put a spammer on the other side and the work stops compounding
Everything above assumed a world that stays still. Spam is not that world. Write a rule about the word free and within a fortnight the mail says fr3e.
Same forty rules, same greedy choices, one change. When a rule lands on a word the spammer is using, they drop it and pick a new one.
The spammer replaced 31 words over forty rounds, and they are not efficient about it. That is enough. The filter's best is 89.5%% against 94.2%% in the quiet world.
The ceiling is not the lesson. The shape is. In the still world early rules stay valuable forever. In the live world each rule buys a little, the ground moves, and the next rule starts from where the last one left off.
Two walls, and rule-based AI ran into both
It gets worse on its own. Past a point, more hand-written knowledge is somebody's local circumstances written down as if they were general truths.
And the bill never stops. Every domain worth automating has something in it that changes: a competitor, a regulation, a fashion, a fraudster.
XCON did not collapse because DEC hired bad engineers. It collapsed because the product line kept changing and the rules had to keep up.
And that is still not the strongest argument
Everything so far assumes the knowledge could be written down, given enough time and people.
The next lesson is about the discovery that broke that assumption.
content/notebooks/introduction-to-ai/the-arms-race-you-lose.ipynb
Change how often the spammer notices, or how many words they have to swap to. Worth trying: let the spammer notice every single time and see whether any number of rules helps.
Show the code5 cells
SPAMMY = [f'spamword{i}' for i in range(40)]
NEUTRAL = [f'word{i}' for i in range(300)]
N, SPAM_RATE, ROUNDS = 8000, 0.40, 40
def make_mail(rng, spam_vocab):
"""One mailbox, written from whatever vocabulary the spammer is using
THIS week. Neither side is pure: plenty of real mail says 'free'."""
mail = []
for _ in range(N):
spam = rng.random() < SPAM_RATE
pool = spam_vocab if spam else NEUTRAL
toks = {rng.choice(pool) for _ in range(rng.randint(4, 9))}
toks |= {rng.choice(NEUTRAL) for _ in range(rng.randint(3, 7))}
if not spam and rng.random() < .18:
toks.add(rng.choice(spam_vocab))
mail.append((toks, spam))
return mail
rng = random.Random(3)
mail = make_mail(rng, SPAMMY)
record('n_mail', f'{len(mail):,}')
record('spam_share', f'{sum(s for _, s in mail)/len(mail):.0%}')
record('vocab_size', len(SPAMMY))# BOTH experiments score on mail the rules were NOT written from.
def score(mail, rules):
return sum((bool(t & rules) == s) for t, s in mail) / len(mail)
def best_next(mail, rules):
vocab = {w for t, _ in mail for w in t} # includes anything new
gain = {}
for w in vocab:
if w in rules: continue
gain[w] = (sum((w in t) and s for t, s in mail)
- sum((w in t) and not s for t, s in mail))
return max(gain, key=gain.get)
BASE = max(SPAM_RATE, 1 - SPAM_RATE)
rng_t = random.Random(99)
test_mail = make_mail(rng_t, SPAMMY) # fresh, never written from
rules, static_curve = set(), []
for _ in range(ROUNDS):
rules.add(best_next(mail, rules))
static_curve.append(score(test_mail, rules))
peak = max(static_curve); peak_at = static_curve.index(peak) + 1
record('static_rule1', f'{static_curve[0]:.1%}')
record('static_rule5', f'{static_curve[4]:.1%}')
record('static_peak', f'{peak:.1%}')
record('static_peak_at', peak_at)
record('static_final', f'{static_curve[-1]:.1%}')
record('do_nothing', f'{BASE:.0%}')# The spammer holds a LIVE vocabulary. When a rule lands on a word it is
# using, it drops that word and invents a brand new one nobody has a rule
# for — VIAGRA becomes V1AGRA. Deliberately imperfect: it notices only 75%
# of the time, and two rounds late. One that adapts instantly and perfectly
# makes the filter never gain a single point, which is a caricature.
rng2 = random.Random(3)
live = list(SPAMMY); fresh = 0; pending = []
rules2, adaptive_curve, replaced = set(), [], 0
for r in range(ROUNDS):
train = make_mail(rng2, live) # this week's mail
w = best_next(train, rules2)
rules2.add(w)
if w in live and rng2.random() < 0.75:
pending.append((r + 2, w)) # noticed, acted on later
for due, word in [x for x in pending if x[0] <= r]:
if word in live:
live.remove(word); fresh += 1
live.append(f'fresh{fresh}'); replaced += 1
pending = [x for x in pending if x[0] > r]
adaptive_curve.append(score(make_mail(rng_t, live), rules2))
apeak = max(adaptive_curve)
record('adaptive_rule1', f'{adaptive_curve[0]:.1%}')
record('adaptive_peak', f'{apeak:.1%}')
record('adaptive_final', f'{adaptive_curve[-1]:.1%}')
record('words_replaced', replaced)
record('gap_at_end', f'{(static_curve[-1]-adaptive_curve[-1])*100:.0f}')
record('n_rules', ROUNDS)def plot(ax):
xs = range(1, ROUNDS+1)
ax.plot(xs, [100*v for v in static_curve], lw=2.2, label='a world that stays still')
ax.plot(xs, [100*v for v in adaptive_curve], lw=2.2, ls='--',
label='a world that answers back')
ax.axhline(100*max(SPAM_RATE, 1-SPAM_RATE), lw=1.1, ls=':',
label='writing no rules at all')
ax.set_xlabel('rules written')
ax.set_ylabel('correct (%)')
ax.set_ylim(40, 102)
ax.legend(frameon=False, loc='lower right', fontsize=9)
save_fig('two-worlds', plot, figsize=(7, 4.2))marg = [static_curve[0]-BASE] + \
[static_curve[i]-static_curve[i-1] for i in range(1, ROUNDS)]
def plot(ax):
ax.bar(range(1, ROUNDS+1), [100*m for m in marg])
ax.set_xlabel('rule number')
ax.set_ylabel('percentage points it added')
save_fig('what-each-rule-bought', plot, figsize=(7, 3.4))
record('marginal_1', f'{100*marg[0]:.1f}')
record('marginal_20', f'{100*marg[19]:.2f}')
record('marginal_40', f'{100*marg[39]:.2f}')