The four steps — named in a line and in a tree
You have built two models four paths apart. A line fitted to 200 markets, and a tree grown on 24 films. They look nothing alike, and the same four questions describe both — except that for the tree, two of them come back empty.
You have built two models. The line fitted a straight line to 200 markets. Growing it, and knowing when to stop grew a tree on 24 films.
They have nothing in common on the page. One is two numbers and a formula. The other is a set of questions and no numbers at all.
Four questions describe both of them. For the tree, two of the four come back empty. This lesson asks the four. The rest of the path is about the two that come back empty.
The line: base sales, plus 47.5 units per extra $1,000 of TV
Two numbers, and a formula that finds them. Where the line starts is 7.03. How fast it rises is 0.0475. Change either one and you have a different line.
That is the whole model. Give it a TV budget and it gives a sales figure back.
The tree: five questions, and a leaf that says hit or flop
No numbers at all, and a shape that grew. Nothing in this tree was set to a value. Its questions were chosen, one at a time, from the data.
Four questions describe both
Ask these of any model, in this order.
- What shape may the answer be? The form. It is chosen before the data is read.
- What is free? The parameters. These are the numbers the data is allowed to set.
- What is it scoring? The cost. One number that says how wrong the model currently is.
- How did it search? The method that found the answer that scores best.
Those four questions are what people mean by fitting a model. The line performed all four of them and named two.
The tree has nothing to put under two of them
| The line | The tree | |
|---|---|---|
| What shape may the answer be? | A straight line | A set of yes or no questions |
| What is free? | Two numbers | — |
| What is it scoring? | Squared error, at 7.03 and 0.0475 | — |
| How did it search? | A formula, straight to the answer | Best question first, then again on each side |
What shape may the answer be?
- The line
- A straight line
- The tree
- A set of yes or no questions
What is free?
- The line
- Two numbers
- The tree
- —
What is it scoring?
- The line
- Squared error, at 7.03 and 0.0475
- The tree
- —
How did it search?
- The line
- A formula, straight to the answer
- The tree
- Best question first, then again on each side
The tree answers the first and the last. There is no parameter, and no cost it ever makes small. Both of those blanks are real, and both of them are measured later in this path.
A tree's leaves do hold numbers — what it lacks is a fixed list of them
Look again at the tree above. Every leaf says how many films it holds and how many of them were hits. Those counts came from the data.
So the blank in that table needs saying carefully. A straight line has exactly two free numbers, whatever data you show it. You can write them down before you fit anything.
A tree's numbers do not exist until its shape does, and its shape grew out of the data. There is no list to write down in advance, and nothing a search could move.
A model with a fixed list of free numbers is called parametric. A model without one is non-parametric. A straight line is the first. A tree is the second.
A line's two numbers move a little. A tree's questions hold or change entirely.
This is what the blank costs in practice. Change a few markets and the line's slope shifts slightly. It is still a line, still two numbers, and both of them moved a small amount.
There is no way to nudge a question. A tree either asks the same one or asks a different one.
The first question held 30 of 30 refits, and 22% of its names still changed
Turning a tree into a list of names measured this on the attrition tree, not the films tree. That one was built on 1,500 employees and refitted 30 times, each time with 5% of the rows changed.
| What was measured | Result |
|---|---|
| Refits where "works overtime" was still the first question | 30 of 30 |
| Change in the top fifty people the tree flagged | 22% |
Refits where "works overtime" was still the first question
- Result
- 30 of 30
Change in the top fifty people the tree flagged
- Result
- 22%
The shape held. What it did with people did not. A model with parameters cannot behave that way, because there is nothing in it that jumps.
Ask the four questions of a model you have never met
The four are worth having because they work on models you have not been taught. Here they are on logistic regression, which is a later path.
| Logistic regression | |
|---|---|
| What shape may the answer be? | A curve that returns a number between 0 and 1 |
| What is free? | One number per input, plus a starting number |
| What is it scoring? | Not squared error. A different cost. |
| How did it search? | Downhill, in steps. There is no formula for the answer. |
What shape may the answer be?
- Logistic regression
- A curve that returns a number between 0 and 1
What is free?
- Logistic regression
- One number per input, plus a starting number
What is it scoring?
- Logistic regression
- Not squared error. A different cost.
How did it search?
- Logistic regression
- Downhill, in steps. There is no formula for the answer.
Three of the four are answerable before you know anything about it. Only the cost is unfamiliar, and that is the one thing that path has to supply.
Two of the four are worth arguing with
The line answered all four, and it is worth asking how settled those answers were.
- The cost was a choice somebody made. The line scored the line by adding up its squared errors and never said why squared. The next lesson tries a different cost on the same data and gets a different line.
- The search stopped where it stopped. The tree took the best question available at each step. The lesson after that builds every tree it could have built instead, and finds one that is better.
Both are places where fitting stops being arithmetic and becomes a decision. Somebody chose the cost. Somebody chose how far the search would look.
Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read