15 min

Cross-validation — the honest number

Every score so far was measured on the markets the model was fitted to, and every one is therefore optimistic. Hold data back to find out by how much. Then find out why holding it back once is not enough.

Here is a fact about every number quoted in this path so far. The R² in The line, the RMSE in Is it any good?, the AIC in Which features earn their place. All of them were computed on the same 200 markets that decided the coefficients.

The model has already seen the answers it is being marked against. That is a large caveat. It is the difference between an exam and an exam you were given the answers to. Any score measured on the data a model was fitted to is optimistic, and the only question is by how much.

Adjusted R² and AIC guessed at the answer from a formula. This lesson measures it.

Fit on some markets, score on the rest

The whole idea takes one line of discipline. Shuffle the markets, fit on some, and score on the rest. The model has never seen those and had no chance to bend itself around them.

Training set — 140 markets. The coefficients are computed from these and nothing else.
Held-out set — 60 markets. Never touched during fitting. Scored once, at the end. This is the exam.

Shuffle, fit on some, score on the rest.

The TV + radio model, scored on both halves of one 70/30 split.a1-regression/the-honest-number.ipynb

The held-out score came out better than the training score, which is the opposite of what the argument predicts.

Fitted on 70% of the markets, scored on both halves.

	RMSE	R²
Training — 140 markets	1.73k units	0.876
Held out — 60 markets	1.57k units	0.914

Training — 140 markets

RMSE: 1.73k units
R²: 0.876

Held out — 60 markets

RMSE: 1.57k units
R²: 0.914

And the held-out score came out better than the training score — 1.57k against 1.73k. Which is the opposite of what the argument above predicted.

Two things are going on, and both are worth more than a clean demonstration would have been. The first is that this model has very little to memorise: two columns and 140 markets means barely enough freedom to bend around anything. The second is that we drew one shuffle out of an enormous number of possible shuffles, and this one handed the model an easy exam. Hold on to that. It is §3.

The held-out score came out better than the training score, which is the opposite of what the argument predicts.

Training R² rose at all 15 steps while held-out R² fell at all 15

To see overfitting you have to give a model enough rope. So bring back the random columns. Take them from Which features earn their place and feed them in one at a time, scoring on both sets after each.

Training score and held-out score as junk columns pile up. Averaged over 20 draws.a1-regression/the-honest-number.ipynb

The two lines go opposite ways, and that is the whole of overfitting in one picture.

The same markets, the same real channels, plus columns of pure noise.

Junk columns	Training R²	Held-out R²	Held-out RMSE
0	0.876	0.914	1.57k
60	0.931	0.832	2.19k

Training R²: 0.876
Held-out R²: 0.914
Held-out RMSE: 1.57k

Training R²: 0.931
Held-out R²: 0.832
Held-out RMSE: 2.19k

The two lines go opposite ways. Training R² rises at 15 of 15 steps and held-out R² falls at 15 of 15. And that is the whole of overfitting in one picture. With 60 junk columns the model has 63 knobs to fit 140 training markets. It uses them to memorise the training set. Training R² climbs to 0.931. And every bit of that memorising makes it worse at markets it has not seen. Held-out R² falls to 0.832 and the held-out miss grows by 0.62k units.

Overfitting is a gap rather than a property of a model. Specifically: the gap between what a model scores on data it has seen and what it scores on data it has not. You cannot see it from inside the training set — which is exactly why the training set cannot be the judge.

Notice what this rescues from the previous lesson. Adjusted R² and AIC did flag the junk, and they were right to. But they were applying a formula for how much optimism to expect. A good formula, derived under assumptions, that knows nothing about your data. The holdout does not assume. It measures.

The two lines go opposite ways, and that is the whole of overfitting in one picture.

The same model on the same markets is worth 1.22k to 2.19k, depending on the shuffle

Now back to that suspiciously friendly result in §1. Deal the cards again — 200 more times, same model, same markets, only the shuffle changing.

The same model, scored 200 times, differing only in which markets landed in the held-out set.a1-regression/the-honest-number.ipynb

So never quote a number that came from one split.

	Held-out RMSE
Luckiest split	1.22k units
Unluckiest split	2.19k units
Average over all 200	1.71k units

Luckiest split

Held-out RMSE: 1.22k units

Unluckiest split

Held-out RMSE: 2.19k units

Average over all 200

Held-out RMSE: 1.71k units

The same model, on the same 200 markets, is worth anything from 1.22k to 2.19k. Nothing changed but which markets fell on which side of the line. A spread of 0.98k units. If you report the number from one split, you are reporting one draw from that range and calling it the truth.

Averaged over all 200 splits, though, the picture the argument predicted does appear: training RMSE averages 1.65k and held-out RMSE averages 1.71k. The optimism of scoring on your own training data is 0.06k units for this model. It is real, consistent and small, because a two-column model on 200 rows has almost nothing to memorise. Add the junk columns and that same gap explodes. The optimism is not a constant; it grows with how much freedom you gave the model.

So never quote a number that came from one split.

Cross-validation holds out every market exactly once

Averaging 200 random splits would work, but it wastes effort and tests some markets many times and others rarely. Cross-validation is the tidy version of the same idea.

Cut the markets into 5 equal folds.
Fit 5 times. Each time, hold one fold out and train on the rest.
Score each held-out fold, then average the scores.

Nothing is wasted, and nothing is scored by a model that saw it.

Every market is used for training 5 − 1 times and scored exactly once, by a model that never saw it. Nothing is wasted and nothing is scored by itself.

Five folds, five held-out scores. The dashed line is the cross-validated RMSE.a1-regression/the-honest-number.ipynb

The folds range from 1.39k to 2.07k. The same lottery as before, in miniature, which is why you average them rather than pick one. The average is 1.69k units, and it lands within 0.02k of the mean over 200 random splits. Five fits got what 200 got.

\mathrm{RMSE}_{\mathrm{CV}} = \frac{1}{K}\sum_{k=1}^{K} \mathrm{RMSE}\big(\text{fold } k \mid \text{model fitted without fold } k\big)

K is usually 5 or 10. Larger K means more fits and less data held out of each.

Where cross-validation quietly lies. It assumes any market could have been any other — that shuffling them is fair. For rows ordered in time it is unfair. Fitting on next quarter to predict last quarter is a way of knowing the future. It hands you a beautiful score for a model that cannot forecast. Time series get a split by date, never a shuffle.

Nothing is wasted, and nothing is scored by a model that saw it.

A score is only honest if the data took no part in any decision

One more trap, and it is the one that catches people who have already learned everything above. If you use your held-out set to choose between models. Try twenty, keep the one that scores best on the holdout. Then the holdout has helped build the model, and its score is optimistic too. You have just overfitted more slowly.

Select with the middle one. Report from the last one.

Set	What it is for	How often it may be used
Training	Fitting the coefficients	constantly
Validation — or cross-validation	Choosing between models: which columns, which settings	as often as you like
Test	Estimating what the chosen model is worth	once, at the end

Training

What it is for: Fitting the coefficients
How often it may be used: constantly

Validation — or cross-validation

What it is for: Choosing between models: which columns, which settings
How often it may be used: as often as you like

Test

What it is for: Estimating what the chosen model is worth
How often it may be used: once, at the end

The number you report should come from data that took no part in any decision. Not the fitting, not the column choice, not the threshold, not the moment you looked at the score and decided to try one more thing. Every one of those is a way the data got to influence the model. Each one buys back some of the optimism you were trying to escape.

1.69k units, about 12% of an average market's sales

The path has now produced three RMSEs for three different models, and only one of them is honest:

Model	RMSE	Measured on
TV only — Is it any good?	3.24k units	the markets it was fitted to
TV + radio — the model Which features earn their place chose	1.67k units	the markets it was fitted to
TV + radio, cross-validated	1.69k units	markets no coefficient ever saw

TV only — Is it any good?

RMSE: 3.24k units
Measured on: the markets it was fitted to

TV + radio — the model Which features earn their place chose

RMSE: 1.67k units
Measured on: the markets it was fitted to

TV + radio, cross-validated

RMSE: 1.69k units
Measured on: markets no coefficient ever saw

1.69k units — about 12% of an average market's sales — is the number to put in front of a marketing lead. It is the first one in this path that answers the question they are asking. Not t "how well does this fit?" but "how wrong will you be about my market next quarter?"

The habit, stated once. Before you quote any score, ask one question: did the model see this data while it was being built? If the answer is yes, or you are not sure, the number is a description of the past and not a forecast of anything.

Run it yourself

Change K from 5 to 10, or to n for leave-one-out cross-validation, and watch how much the answer moves. The interesting result is how little. Then re-run the junk-column experiment with MAX_JUNK = 120 and watch the model become a perfect memoriser and a useless forecaster.

The honest number — holdouts and cross-validation

content/notebooks/a1-regression/the-honest-number.ipynb

Download Open in Colab

Splits the markets, scores both halves, manufactures overfitting with junk columns, shows the single-split lottery across 200 shuffles, cross-validates, and ends by quoting one plan's forecast with a range built from the cross-validated error. Runs unchanged in Colab.

Show the code5 cells

Fit on 70% of the markets, score on the 30% never seen

import numpy as np
import pandas as pd

df = pd.read_csv(CSV)
y = df['sales'].to_numpy()
n = len(df)
X = df[['tv_spend', 'radio_spend']].to_numpy()   # the model L6 chose

def fit_and_score(X, train, test):
    """Fit on the training rows only, then score on BOTH sets.

    The whole discipline of this lesson is in one line: `coef` is computed from
    X[train] and never sees X[test].
    """
    design = lambda rows: np.column_stack([np.ones(len(rows)), X[rows]])
    coef, *_ = np.linalg.lstsq(design(train), y[train], rcond=None)

    def scores(rows):
        err = y[rows] - design(rows) @ coef
        rmse = float(np.sqrt((err ** 2).mean()))
        r2 = 1 - (err ** 2).sum() / ((y[rows] - y[rows].mean()) ** 2).sum()
        return rmse, float(r2)

    return scores(train), scores(test)

TRAIN_SHARE = 0.7
cut = int(TRAIN_SHARE * n)
order = np.random.default_rng(42).permutation(n)
train, test = order[:cut], order[cut:]

(train_rmse, train_r2), (test_rmse, test_r2) = fit_and_score(X, train, test)
print(f'fitted on {len(train)} markets, scored on {len(test)} it never saw')
print(f'  training   RMSE {train_rmse:.3f}k   R2 {train_r2:.3f}')
print(f'  held out   RMSE {test_rmse:.3f}k   R2 {test_r2:.3f}')

Training score against held-out score, as junk columns pile up

MAX_JUNK, DRAWS = 60, 20
rng = np.random.default_rng(5)
junk = rng.standard_normal((DRAWS, MAX_JUNK, n))

steps = list(range(0, MAX_JUNK + 1, 4))
curve = {'train_r2': [], 'test_r2': [], 'train_rmse': [], 'test_rmse': []}
for j in steps:
    runs = [fit_and_score(np.column_stack([X] + [junk[d, k] for k in range(j)]), train, test)
            for d in range(DRAWS)]
    curve['train_rmse'].append(float(np.mean([tr[0] for tr, _ in runs])))
    curve['train_r2'].append(float(np.mean([tr[1] for tr, _ in runs])))
    curve['test_rmse'].append(float(np.mean([te[0] for _, te in runs])))
    curve['test_r2'].append(float(np.mean([te[1] for _, te in runs])))

print(f"{'junk':>5s} {'train R2':>9s} {'held-out R2':>12s} {'held-out RMSE':>14s}")
for i, j in enumerate(steps):
    if j % 12 == 0:
        print(f"{j:5d} {curve['train_r2'][i]:9.3f} {curve['test_r2'][i]:12.3f} "
              f"{curve['test_rmse'][i]:13.2f}k")

The same model, 200 different 70/30 splits

N_SPLITS = 200
trains, tests = [], []
for seed in range(N_SPLITS):
    o = np.random.default_rng(seed).permutation(n)
    (tr_rmse, _), (te_rmse, _) = fit_and_score(X, o[:cut], o[cut:])
    trains.append(tr_rmse)
    tests.append(te_rmse)
trains, tests = np.array(trains), np.array(tests)

print(f'held-out RMSE over {N_SPLITS} splits:')
print(f'  best  {tests.min():.2f}k     worst {tests.max():.2f}k     mean {tests.mean():.2f}k')
print(f'average training RMSE {trains.mean():.2f}k  vs  average held-out RMSE {tests.mean():.2f}k')

Five-fold cross-validation

K = 5
folds = np.array_split(np.random.default_rng(0).permutation(n), K)

fold_rmse = []
for k in range(K):
    held = folds[k]
    rest = np.concatenate([folds[j] for j in range(K) if j != k])
    _, (rmse, _) = fit_and_score(X, rest, held)
    fold_rmse.append(rmse)
    print(f'fold {k + 1}: held out {len(held)} markets, RMSE {rmse:.2f}k')

print(f'\ncross-validated RMSE: {np.mean(fold_rmse):.2f}k '
      f'(spread {min(fold_rmse):.2f}–{max(fold_rmse):.2f})')

The shipped model, and what it says about one plan

design = np.column_stack([np.ones(n), X])
ship, *_ = np.linalg.lstsq(design, y, rcond=None)
base, per_tv, per_radio = ship

PLAN_TV, PLAN_RADIO = 150.0, 30.0        # both inside the range the model has seen
forecast = base + per_tv * PLAN_TV + per_radio * PLAN_RADIO
margin = 2 * np.mean(fold_rmse)          # ~95% of markets, if the errors behave

print(f'base sales           {base:6.2f}k units')
print(f'per $1k of TV        {per_tv:+.4f}k units')
print(f'per $1k of radio     {per_radio:+.4f}k units')
print(f'\nplan: ${PLAN_TV:.0f}k TV + ${PLAN_RADIO:.0f}k radio')
print(f'forecast             {forecast:6.2f}k units')
print(f'honest range         {forecast - margin:.2f}k to {forecast + margin:.2f}k '
      f'(±2 × cross-validated RMSE)')

Next: the last lesson has no new technique in it. It takes everything this path produced. A model, a coefficient per channel, and a number for how wrong it will be. And turns it into the thing that was wanted in the first place: a recommendation about next quarter's budget that survives being argued with.

Try this in Expedify →

Related lessons

12 min read

Base rates — what a piece of evidence is actually worth

A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.

Read

10 min read

Confirmation and survivorship — what you never looked for

Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.

Read

14 min read

Loss aversion, sunk cost and regression — what it costs you

Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.

Read