Feature selection — which features earn their place
R² cannot go down when you add a column. So it cannot decide which columns to keep. Watch it climb on columns of pure random numbers, then meet the two scores that charge a fee for every column: adjusted R² and AIC.
We found the dead channel by looking at it. Three columns, one obvious collapse, and More than one thing matters could spend a whole lesson on it.
That does not scale. A real media file has spend by channel by region by week, plus price, plus distribution, plus a competitor index. Forty columns, and no amount of squinting will tell you which twelve belong in the model. You need a rule.
The obvious rule is: keep the columns that make the model better. Applied with R², that rule keeps every column you ever try. Here is why.
R² rose at all 15 steps as columns of pure nonsense were added
This is not a tendency. It is arithmetic, and it follows from what least squares does.
Adding a column gives the fitting method one more knob. It can always set that knob to zero and reproduce the old model exactly. So the new model can never fit worse than the old one. And it never settles for equal. With a finite number of markets there is always some accidental alignment between a new column and the error left over. Least squares finds it and pockets it.
So take the full three-channel model and start adding columns of random numbers — numbers with no connection to sales whatsoever, invented after the fact.
It climbs at 15 of 15 steps: 0.89721 with the three real channels, 0.90491 once 15 columns of nonsense have joined them. A gain of 0.0077 bought with nothing at all — against the 0.000016 the real newspaper budget managed in More than one thing matters.
This is the promise made in the lesson on scoring, now paid. "Add any column at all — even random noise — and R² goes up, never down." You have just watched it happen 15 times. A score that improves when you add nonsense cannot be used to choose what to add.
85% of random columns bought more R² than the newspaper budget did
Not zero — that is the surprise. A junk column is worth whatever it mops up by luck, and luck is worth a predictable amount. So put a number on it: draw 500 random columns, add each one to the TV + radio model, and record what each buys.
Read that twice. Invented numbers beat the newspaper budget.
| R² gained | |
|---|---|
| The median random column | 0.000258 |
| The luckiest of the 500 | 0.005101 |
| The real newspaper budget | 0.000016 |
The median random column
- R² gained
- 0.000258
The luckiest of the 500
- R² gained
- 0.005101
The real newspaper budget
- R² gained
- 0.000016
85% of the random columns bought more R² than the newspaper budget did. Read that twice. In this data, newspaper is it is worse than noise. A column of numbers drawn from thin air would have flattered the model more than 15% of the media plan did.
And the price of noise is not a mystery. The variation still unexplained has to be shared out among the degrees of freedom that are left, so on average one junk column is worth:
For this model that is 0.000525, and the 500 draws averaged 0.000520. Theory and experiment agree to the fifth decimal — which matters, because that number is about to become a formula.
Read that twice. The newspaper budget is beaten by invented numbers.
Adjusted R² charges every column the price a junk column would pay
If a useless column buys 0.000525 of R², then the fix is to charge every column exactly that much on the way in. What survives the fee is what was doing more than noise would have.
Every extra column shrinks the denominator, which is the fee.
Run the same junk-column experiment and score it both ways:
Across 15 junk columns R² gains 0.0077 and adjusted R² moves -0.00018 — flat, to within a rounding error. That is the fee working as designed. A column that buys the going rate for noise pays the going rate for noise, and nets out at nothing.
Which means adjusted R² can now answer the question that started this path:
| Model | R² | Adjusted R² | AIC |
|---|---|---|---|
| TV + radio | 0.89719 | 0.89615 | 212.8 |
| TV + radio + newspaper | 0.89721 | 0.89564 | 214.8 |
TV + radio
- R²
- 0.89719
- Adjusted R²
- 0.89615
- AIC
- 212.8
TV + radio + newspaper
- R²
- 0.89721
- Adjusted R²
- 0.89564
- AIC
- 214.8
R² goes up by 0.000016, as it must. Adjusted R² goes down by 0.00051. Newspaper did not clear the bar a random column clears, so the fee costs more than the column brought in. And for the first time a score has told us to leave something out.
Say it as a sentence, because the formula hides it. Adjusted R² asks: did this column do more for the model than a column of random numbers would have? That is a fair question to ask of a media channel, a lead score, or a feature somebody is proud of.
What survives the fee is what was doing real work.
AIC asks the same question in a form that survives leaving regression
Adjusted R² is tied to R², which is tied to squared error, which is tied to linear regression. AIC asks the same question in a form that survives leaving all of that behind. It works for logistic regression, for time series, for anything fitted by likelihood.
The shape is the same trade in different clothes. The first term falls as the fit improves, the second rises by a fixed toll of 2 per parameter. What differs is how hard it charges. Over the same 15 junk columns, adjusted R² barely flinched (-0.00018) while AIC rose from 214.8 to 229.1. AIC does not merely refuse to reward junk. It punishes it, which is what you want from something making the decision rather than describing it.
So score every model you could have built. Three channels make 8:
Every model these three channels can make, ranked.
- The winner is TV + radio at AIC 212.8. The runner-up is TV + radio + newspaper — the same model plus newspaper — 2 worse for the extra column.
- R² would have picked TV + radio + newspaper. It always picks the biggest model on offer, which is the whole problem.
- Adjusted R² picks TV + radio, agreeing with AIC. On a question this small they usually will; the disagreements start when models differ by many columns at once.
Reading an AIC gap. The number itself is meaningless. It moves if you change the units of sales. Only the difference between two models fitted to the same rows carries information. The working convention: a gap under about 2 is not worth arguing about, and 10 or more is decisive. Newspaper's 2 is right at that lower edge: the evidence says drop it, and it does not say so loudly. What makes the case here is not the gap. It is the gap plus a coefficient of nearly zero plus 85% of random columns doing better.
It works for logistic regression, for time series, for anything fitted by likelihood.
Three things none of these scores can do
You now have two defensible ways to choose between models. Both have hard edges, and all three of these are worth more than the formulas.
- They cannot compare models fitted on different rows. Drop the markets where a column is missing and you have changed the data, not just the model. The errors are being summed over a different set. An AIC or an R² compared across two row sets is not a comparison at all. Same rows, or no comparison.
- They cannot tell you a column will keep working. Every score here is computed on markets that already happened. A channel that earned its place last year can stop earning it, and nothing in this lesson would notice.
- They are still measured on the data the model was fitted to. Every number on this page — R², adjusted R², AIC — was computed on the same 200 markets that decided the coefficients. The fee is a guess at how much optimism to subtract. It is a good guess, from a formula, made without ever testing the model on anything it had not already seen.
All three are worth more than the formulas.
How to actually choose
- Start from the decision, not the columns. A channel nobody can move the budget on does not need to be in a budget model, however well it scores.
- Never let R² cast a vote. Quote it to describe a model. Never use it to compare two.
- Compare on adjusted R² or AIC — same rows, small number of candidates. With three channels you can score all 8 models. With forty columns you cannot, and the automatic search procedures that promise to are their own kind of trouble, which the regularisation path takes up.
- Then check the coefficient, not just the score. Newspaper's case was closed by its coefficient sitting at zero and by 85% of random columns beating it. The AIC gap only confirmed it.
Two defensible scores, and a decision that still belongs to a person.
Run it yourself
The score() helper takes columns or raw arrays, so you can hand it anything. Change MAX_NOISE to 100 and watch R² pass 0.95 on a model that knows nothing. And watch what AIC does to you for it.
content/notebooks/a1-regression/which-features-earn-their-place.ipynb
Adds columns of random numbers to prove R² can only rise, prices what a junk column is worth, then scores all eight possible models on adjusted R² and AIC. Runs unchanged in Colab.
Show the code4 cells
import numpy as np
import pandas as pd
df = pd.read_csv(CSV)
y = df['sales'].to_numpy()
n = len(df)
def score(columns):
"""Least squares on any list of columns (names or arrays). Three scores back.
R2 share of variation explained
adj R2 charged a fee per column: 1 - (1-R2)(n-1)/(n-p-1)
AIC n*ln(RSS/n) + 2k, k = p + 2 (the slopes, the intercept, the error variance).
Only DIFFERENCES in AIC mean anything, so the constant is left out.
"""
cols = [df[c].to_numpy() if isinstance(c, str) else c for c in columns]
X = np.column_stack([np.ones(n)] + cols)
coef, *_ = np.linalg.lstsq(X, y, rcond=None)
rss = float(((y - X @ coef) ** 2).sum())
p = len(cols)
r2 = 1 - rss / ((y - y.mean()) ** 2).sum()
adj = 1 - (1 - r2) * (n - 1) / (n - p - 1)
aic = n * np.log(rss / n) + 2 * (p + 2)
return dict(r2=r2, adj=adj, aic=aic, p=p)
LADDER = {
'TV': ['tv_spend'],
'TV + radio': ['tv_spend', 'radio_spend'],
'TV + radio + newspaper': ['tv_spend', 'radio_spend', 'newspaper_spend'],
}
print(f"{'model':24s} {'R2':>8s} {'adj R2':>9s} {'AIC':>8s}")
for label, columns in LADDER.items():
s = score(columns)
print(f"{label:24s} {s['r2']:8.5f} {s['adj']:9.5f} {s['aic']:8.2f}")DRAWS, MAX_NOISE = 200, 15
rng = np.random.default_rng(3)
noise = rng.standard_normal((DRAWS, MAX_NOISE, n))
CHANNELS = ['tv_spend', 'radio_spend', 'newspaper_spend']
curve = {'r2': [], 'adj': [], 'aic': []}
for j in range(MAX_NOISE + 1):
runs = [score(CHANNELS + [noise[d, k] for k in range(j)]) for d in range(DRAWS)]
for key in curve:
curve[key].append(float(np.mean([r[key] for r in runs])))
print(f"{'junk cols':>9s} {'R2':>9s} {'adj R2':>9s} {'AIC':>8s}")
for j in (0, 1, 5, 10, 15):
print(f"{j:9d} {curve['r2'][j]:9.5f} {curve['adj'][j]:9.5f} {curve['aic'][j]:8.2f}")N_RANDOM = 500
rng = np.random.default_rng(11)
base = ['tv_spend', 'radio_spend']
r2_base = score(base)['r2']
gains = np.array([score(base + [rng.standard_normal(n)])['r2'] - r2_base
for _ in range(N_RANDOM)])
gain_newspaper = score(base + ['newspaper_spend'])['r2'] - r2_base
# A useless column is not worth zero — it is worth the variation it mops up by
# luck, and that has a closed form: the variation still unexplained, shared out
# over the degrees of freedom left. Print both to see they agree.
expected_junk = (1 - r2_base) / (n - 3 - 1)
print(f'newspaper adds {gain_newspaper:.6f} to R²')
print(f'a random column adds {np.median(gains):.6f} at the median, '
f'{gains.mean():.6f} on average, up to {gains.max():.6f}')
print(f'theory says a junk column is worth {expected_junk:.6f}')
print(f'random columns that beat newspaper: {(gains > gain_newspaper).mean():.1%}')from itertools import combinations
subsets = [c for r in range(4) for c in combinations(CHANNELS, r)]
ranked = sorted(((score(list(c)), c) for c in subsets), key=lambda pair: pair[0]['aic'])
PRETTY = {'tv_spend': 'TV', 'radio_spend': 'radio', 'newspaper_spend': 'newspaper'}
name = lambda combo: ' + '.join(PRETTY[c] for c in combo) or '(nothing)'
print(f"{'model':32s} {'R2':>8s} {'adj R2':>9s} {'AIC':>8s}")
for s, combo in ranked:
print(f"{name(combo):32s} {s['r2']:8.4f} {s['adj']:9.4f} {s['aic']:8.1f}")Next: every score in this lesson was measured on the markets the model was fitted to, and every one of them is therefore optimistic. Next is how much. You find that out by keeping data back. It is the difference between a number you quote and a number you would bet on.
Related lessons
Base rates — what a piece of evidence is actually worth
A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.
ReadConfirmation and survivorship — what you never looked for
Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.
ReadLoss aversion, sunk cost and regression — what it costs you
Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.
Read