Module · Framing

R², RMSE and residuals — is it any good?

Lesson 4 of 8 · 13 min

The line produced a line and an R² of 0.612. Somebody is about to ask whether that is good, and "sixty-one percent" is not an answer to any question they have.

There are three standard ways to score a regression. They measure genuinely different things, and only two are in units anybody can act on. None of them tells you the thing that matters most: whether the model is wrong in a pattern. That last question needs a picture, and it is the second half of this lesson.

R² compares your model against quoting the average

R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}

The model's error, over the error of predicting the average every time.

Read the denominator. It is the error you would make by ignoring TV and quoting the average for every market. That is the flat line The line scored first. So R² does not measure accuracy. It measures how much better you did than the laziest possible model.

Ours is 0.612: TV explains about three-fifths of why markets differ from one another. That is a real answer to a real question. It is just not the question "can I plan with this?"

R² is also the easiest number to inflate. Add any column at all — even random noise — and R² goes up, never down. That is not a hypothetical. It is what the feature-selection lesson later in this path is about, and it is why R² alone should never decide between two models.

The model is typically off by 23% of average sales

These two answer the planning question, in the units of the thing you are predicting.

\mathrm{RMSE} = \sqrt{\frac{1}{n}\sum (y_i - \hat{y}_i)^2} \qquad \mathrm{MAE} = \frac{1}{n}\sum \lvert y_i - \hat{y}_i \rvert

Same errors, two ways of averaging them.

Same errors, two ways of averaging them.

	Value	What it says
RMSE	3.24k units	typical miss, with big misses weighted heavily
MAE	2.55k units	typical miss, every market counted once
RMSE ÷ MAE	1.27	above 1 means a few large misses are doing the damage
RMSE as % of average sales	23%	average sales are 14.02k units

RMSE

Value: 3.24k units
What it says: typical miss, with big misses weighted heavily

MAE

Value: 2.55k units
What it says: typical miss, every market counted once

RMSE ÷ MAE

Value: 1.27
What it says: above 1 means a few large misses are doing the damage

RMSE as % of average sales

Value: 23%
What it says: average sales are 14.02k units

Now put the two descriptions of this model side by side. It explains 0.612 of the variation, and it is typically off by 23% of average sales. Both are true. Only the second one tells a marketing lead what happens when they plan with it.

How far off, market by market. MAE dashed, RMSE solid.a1-regression/is-it-any-good.ipynb

61% of markets fall inside the MAE, which is what you would expect. The tail on the right is why RMSE sits further out. Squaring means one market missed by 8k units counts for more than four missed by 2k.

Which one to quote? Quote RMSE, in units, when somebody will plan against the number. It is more honest about the bad cases, and the bad cases are what break plans. MAE when you are comparing models and want a stable, interpretable average. Quote R² only alongside one of them, never alone.

RMSE punishes big misses. MAE treats every miss the same.

Only 25% of markets are predicted within 1,000 units

Every market, plotted against what the model said. The diagonal is perfection.

Actual against predicted. The dashed line is where a perfect model's points would sit.a1-regression/is-it-any-good.ipynb

Only 25% of markets are predicted within 1,000 units, and 102 of the 200 are off by more than 2,000. Held against the 0.612 we started with, that is the whole lesson. A model can explain most of the variation and still be too coarse to plan a single market with.

The residuals arc and fan, so the model is missing something

Here is the plot worth more than all three scores, and the reason it comes last rather than first.

A score is a single number, so it can only tell you how much you are wrong. The residual plot tells you where and in what direction. A model that is wrong in a pattern is missing something you could still go and find. If the model has caught everything there is to catch, what is left is noise: a shapeless band around zero.

Error against prediction. The line is the local average error — flat would mean nothing left to find.a1-regression/is-it-any-good.ipynb

A flat, shapeless band would mean nothing was left to find. This is not that.

Two shapes are visible, and Look before you fit predicted both.

It arcs. Take the model's average error in each third of the TV budgets. Across the lowest third it errs by -0.07k, across the middle third by +0.26k, and across the top third by -0.19k. Under-predicting in the middle and over-predicting at both ends is the signature of a straight line fitted to a curve. It is the diminishing returns the scatter showed in Look before you fit, now measured.
It fans. Error spread is 1.86k units among low-budget markets and 4.42k among high-budget ones — nearly two and a half times wider. The model is least reliable exactly where the money is. A ±1.86k forecast on a small market is useful; a ±4.42k forecast on your biggest market is barely a forecast.

This is the habit to take away from this lesson. Plot the residuals before you quote the score. A flat, shapeless band means the score can be trusted at face value. Any curve, any fan, any drift means the number is an average over cases the model treats very differently. The average hides which ones.

A flat, shapeless band would mean there was nothing left to find. This is not that.

The worst miss promised 20.19k units and the market sold 11.8k

Scores are abstractions. A market you got badly wrong is a plan somebody built and defended.

The eight biggest misses. Bars to the left mean the model promised more than the market delivered.a1-regression/is-it-any-good.ipynb

The worst is a market that spent $276.7k on TV. The model predicted 20.19k units; it sold 11.8k — short by 8.39k units. Anyone who built a plan on that forecast committed to inventory, staffing and a revenue number for roughly twice what arrived.

That is not a failure of the technique. It is a heavy TV budget in a market where something else was working against it. Competition, distribution, or a channel not in this file. Which is the correct reading of a large residual: not "the model is bad", but "something is going on here that the model cannot see". Chasing those cases is how the next feature gets found.

Good as an explanation, not good enough to plan one market on

As an explanation, yes. TV genuinely drives sales, and 0.612 of the variation between markets is accounted for by one column.
As a planning tool for one market, no. Typically off by 23%, and worst exactly where budgets are largest.
And it is wrong in a pattern, which is good news. The arc and the fan are both fixable. The arc goes with a curve, the fan by modelling on a different scale. A great deal of the remaining error goes with the two channels still unused in the file.

All three of those are true at once, and only the second one stops a plan.

Run it yourself

Recompute all three scores, then change thirds to five bands instead of three and watch the arc get clearer.

Is it any good — three scores and a residual plot

content/notebooks/a1-regression/is-it-any-good.ipynb

Download Open in Colab

Refits the previous lesson's line, computes R², RMSE and MAE, plots the errors four ways, and ranks the worst markets. Runs unchanged in Colab.

Show the code5 cells

Fit the same line as lesson 3

import numpy as np
import pandas as pd

df = pd.read_csv(CSV)
x, y = df['tv_spend'], df['sales']

slope, intercept = np.polyfit(x, y, 1)
predicted = intercept + slope * x
error = y - predicted          # positive = we under-predicted this market

print(f'sales = {intercept:.2f} + {slope:.4f} x TV')

R², RMSE and MAE

r2   = 1 - (error ** 2).sum() / ((y - y.mean()) ** 2).sum()
rmse = np.sqrt((error ** 2).mean())
mae  = error.abs().mean()

print(f'R2   = {r2:.3f}   share of the variation in sales the model explains')
print(f'RMSE = {rmse:.2f}k units   typical miss, big misses weighted heavily')
print(f'MAE  = {mae:.2f}k units   typical miss, every market counted equally')
print(f'mean sales = {y.mean():.2f}k units  <- RMSE is {100 * rmse / y.mean():.0f}% of it')

How far from the diagonal?

on_the_nose = (error.abs() <= 1.0).mean()      # within 1,000 units
print(f'{100 * on_the_nose:.0f}% of markets predicted within 1,000 units')
print(f'worst market: off by {error.abs().max():.2f}k units')

Is the error random, or does it have a shape?

thirds = pd.qcut(x, 3, labels=['low TV', 'mid TV', 'high TV'])
bias = error.groupby(thirds, observed=True).mean()

for band, mean_error in bias.items():
    direction = 'under-predicts' if mean_error > 0 else 'over-predicts'
    print(f'{band:8s}: average error {mean_error:+.2f}k  ->  {direction}')

The eight worst markets

worst = (pd.DataFrame({'tv': x, 'actual': y, 'predicted': predicted, 'error': error})
         .assign(miss=lambda d: d.error.abs())
         .nlargest(8, 'miss')
         .round(2))
worst[['tv', 'actual', 'predicted', 'error']]

Next: the two channels we have been ignoring. Radio and newspaper go into the model and R² jumps. One of those two channels turns out to have been doing nothing at all.