The five approaches to AI — where the knowledge comes from
Five serious ideas about how to build a machine that decides. They are usually taught as a chronology, one retiring the last. That story is tidy and wrong, and the correction is what the second half of this course rests on.
Open your phone and something is already deciding for you. Which video plays next. Whether that payment goes through. How long your food will take.
There have been five serious ideas about how to build a machine that can decide. They are usually taught as a chronology, each one retiring the last.
That story is tidy, memorable and wrong. The correction is what the second half of this course rests on.
One question sorts the whole field: where does the knowledge come from?
The five differ in a hundred technical ways, and in exactly one that matters to you.
Where does the knowledge come from?
An order going out in heavy rain, eight kilometres away, at 8pm on a Friday, is going to be late. Somebody, or something, has to know that. The five approaches are five different answers to who that somebody is.
Nothing further down that list is more intelligent than anything above it. What changes is who did the hard part.
Every one of the five has something running on your phone right now
| Approach | Where the knowledge comes from | Something near you running on it |
|---|---|---|
| Rule-based | A person writes it down, one rule at a time | Whether your coupon applies, your card's daily limit, whether you are old enough to buy that |
| Search & planning | A person describes the world and the goal; the machine explores the possibilities | Maps finding your route, the rider assigned to your order, chess and Go engines |
| Probabilistic | A person states a starting belief; data revises it | The spam in your inbox, the fraud check on your card |
| Machine learning | A person supplies examples with answers; the machine writes the rules | Your delivery ETA, your credit score, the price of a cab in the rain |
| Deep learning | A person supplies raw data; the machine works out the useful features too | Face unlock, voice notes turned into text, ChatGPT, your photos app finding your friend |
Rule-based
- Where the knowledge comes from
- A person writes it down, one rule at a time
- Something near you running on it
- Whether your coupon applies, your card's daily limit, whether you are old enough to buy that
Search & planning
- Where the knowledge comes from
- A person describes the world and the goal; the machine explores the possibilities
- Something near you running on it
- Maps finding your route, the rider assigned to your order, chess and Go engines
Probabilistic
- Where the knowledge comes from
- A person states a starting belief; data revises it
- Something near you running on it
- The spam in your inbox, the fraud check on your card
Machine learning
- Where the knowledge comes from
- A person supplies examples with answers; the machine writes the rules
- Something near you running on it
- Your delivery ETA, your credit score, the price of a cab in the rain
Deep learning
- Where the knowledge comes from
- A person supplies raw data; the machine works out the useful features too
- Something near you running on it
- Face unlock, voice notes turned into text, ChatGPT, your photos app finding your friend
Read the third column before the second. Every row has something running on your phone right now.
The knowledge stops being sentences and starts being numbers
That word is doing a lot of work, so here it is made literal. One question — will this delivery be late? — and the thing that holds the answer in each of the five.
# RULE-BASED — a person wrote this sentence and meant it
if distance_km > 8 and raining:
return "late"
# SEARCH — a person wrote the goal and the map, not the answer
goal = "reach the customer in the fewest minutes"
moves = ["main road", "back lane", "flyover", "wait for the light"]
# ...the machine tries routes until one is best
# PROBABILISTIC — a person wrote the starting belief; data moves it
P(late) = 0.12 # what we believed before looking outside
P(late | raining) = 0.34 # what the data revised it to
# MACHINE LEARNING — nobody wrote this. It was fitted to 40 million orders.
weights = {"distance": 0.41, "orders_in_queue": 0.77, "rider_rating": -0.30}
# DEEP LEARNING — nobody wrote this either, and nobody named the columns
layer_1 = [[0.02, -0.71, ...], [0.55, 0.13, ...], ...] # 12,288 numbersThe top of that block can be read, questioned and corrected by a person. The bottom cannot. What you buy on the way down is the ability to handle problems nobody could write rules for. What you pay is that you can no longer read why it decided.
Learning took over because of what it costs to get better
Not because it is cleverer.
- Hand-written knowledge costs a person per rule. Twice as good means roughly twice as many rules, each written, tested and maintained by somebody who understands the problem.
- Learned knowledge costs data. Twice as good often just means more examples. A delivery app generates forty million of them a month whether anyone uses them or not.
The two scale differently, and one of them scales in the direction the modern world happens to point. That is the honest reason machine learning won.
Order dinner tonight and all five approaches fire off one tap
Read the spectrum again and it is tempting to see a relay race. Rules hand off to search, search to statistics, and everybody else goes home. Nobody went home.
- Every generation is still working. The oldest idea in the field is deciding your discount at the same instant the newest one is ranking your feed.
- The famous one is not in charge of the money. Deep learning chooses what you see. A hand-written rule decides what you are allowed to be given.
This is the architecture Track 2 builds. If the eras story were true, that track would be nonsense. It makes complete sense once you see the approaches as a toolbox rather than a queue.
Where the rest of this course sits on the map
- The rest of this path — the first two. Rules and search, taken seriously and then pushed until they break. That break is why the other three exist.
- The rest of AI & ML Foundations — mostly the fourth. Machine learning: trees, regression, classification, ensembles, clustering.
- Later tracks — the fifth. Deep learning, language models, and the mathematics underneath all of it.
- The third, probability, threads through all of them rather than getting a home of its own.
Next we take the first one seriously. A rule-based system written by somebody who knows the business, which genuinely works. And then the wall it hits.
Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read