12 min

The cost function — squared error is a decision, not a law

One number has been falling for a whole path and nobody has said where it comes from. Squared error was a decision somebody made. Make a different one and the same data gives you a different line.

The line scored a line by adding up its squared errors. It never said why squared.

That is worth stopping on. The cost is the thing the fitting makes small, so whatever it measures is what the model is built to be good at.

Change the cost and you change the model, with the same data and the same straight line. This lesson does exactly that.

Cost function, loss function and objective function are one idea

Three names arrive for this, here and everywhere else. They mean one thing: a number that says how wrong the current model is.

A cost function takes a model and returns one number. Fitting makes that number small. This path says cost. You will read the other two in the same sentence as this one elsewhere.

One distinction is worth carrying. An objective is sometimes made large rather than small. Information gain is one, and Which question to ask first had you maximising it.

Making a cost small and making a gain large are the same act with a minus sign in front.

The cost is built from the gap between each point and the line

Every market sold something. The line predicts something else. The difference between them is that market's gap.

One stroke per market. The longest is 8.39, and the middle one is 2.03.how-a-model-is-fitted/the-cost-is-a-choice.ipynb

Two hundred gaps. One number. The cost is whatever turns that first thing into the second, and there is more than one way to do it.

Square the gaps, or just add their sizes. Both are honest.

Here are the two. Each one averages over all 200 markets.

\text{squared error} = \frac{1}{n}\sum_{i=1}^{n} \bigl(y_i - (b_0 + b_1 x_i)\bigr)^2

Square each gap, add them up, divide by the number of markets.

Squaring is one way to stop a gap above the line cancelling a gap below it. Here is another.

\text{absolute error} = \frac{1}{n}\sum_{i=1}^{n} \bigl| y_i - (b_0 + b_1 x_i) \bigr|

Take the size of each gap, add them up, divide by the number of markets.

Both give one number, and both are smallest when the line sits in the middle of the cloud. Neither one lets a gap above cancel a gap below.

One practical difference. Squared error has a formula that lands straight on the answer, which is what The line used. Absolute error has none, so it is fitted by searching.

The two costs pick different lines: 0.04754 and 0.05063

Fit the same straight line to the same 200 markets under each cost.

Same data, same form, same search for the smallest number. Two answers.how-a-model-is-fitted/the-cost-is-a-choice.ipynb

Nothing changed except what wrong was taken to mean. The slope is what a marketer would act on, and the two costs disagree about it.

Each cost ranks its own line first

Score both lines under both costs. There are only four numbers, and they settle nothing.

Lower is better. Read each row across, then each column down.

	Squared error	Absolute error
The line squared error picks (slope 0.0475)	10.513	2.550
The line absolute error picks (slope 0.0506)	10.585	2.534

The line squared error picks (slope 0.0475)

Squared error: 10.513
Absolute error: 2.550

The line absolute error picks (slope 0.0506)

Squared error: 10.585
Absolute error: 2.534

Each cost puts its own line first. There is no view from outside the two that says which line is better, because better is the thing the cost was defining.

One odd market moves the squared line 16 times as far

So the choice has to be made on what it does. Add one market to the file: a TV budget of $10 thousand, and 26 thousand units sold. Then refit both lines.

One market added to 200. Both lines refitted.how-a-model-is-fitted/the-cost-is-a-choice.ipynb

The squared line moved 0.00170. The absolute line moved 0.00011.

The same added market, and what each cost did about it.

	Slope before	Slope after	Moved by
Squared error	0.0475	0.0458	0.0017
Absolute error	0.0506	0.0505	0.0001

Squared error

Slope before: 0.0475
Slope after: 0.0458
Moved by: 0.0017

Absolute error

Slope before: 0.0506
Slope after: 0.0505
Moved by: 0.0001

The reason is arithmetic. That market sits 18.5 away from the old line, and squared error counts that gap as 342. Absolute error counts it as 18.5.

So squared error is the choice to fear big misses

A cost that squares its gaps will accept many small errors to avoid one large one. A cost that does not will leave the large one alone.

Squared error, when a big miss is expensive. Forecasting stock for one warehouse, where being 500 units short once costs more than being 20 units off twenty times.
Absolute error, when every unit of error costs the same. And when the data has a few strange rows you have decided not to let dominate the answer.

Squared error is the standard choice, and it is still a choice. It is the default in every regression tool you will use, which is exactly why it is worth knowing what it assumes.

Ask what a model is scoring before you ask how it scored

A score means nothing until you know what it was measuring. Perfect on what it has seen made a harder version of the same point, where accuracy put three models in the wrong order.

Only hit or miss made the first version of it. A single number of feedback is a cost function under another name, and what it counts decides what gets learned.

That is step three of the four. The next lesson takes step four, and the model that does it differently.

Two costs, two lines, and one added market

content/notebooks/how-a-model-is-fitted/the-cost-is-a-choice.ipynb

Download Open in Colab

Move the added market. Push it further out along the TV axis and both lines follow it; push it further up in sales and only the squared one does. Worth trying: fit a cost that raises each gap to the fourth power and watch how little else it listens to.

Show the code6 cells

The same 200 markets a1-regression used

import numpy as np
import pandas as pd
from scipy.optimize import minimize

df = pd.read_csv(CSV)
x = df['tv_spend'].to_numpy()   # TV budget, $ thousands
y = df['sales'].to_numpy()      # sales, thousands of units
record('n_markets', len(y))

def mse(a, b, X=x, Y=y):
    return float(np.mean((Y - (a + b * X)) ** 2))

def mae(a, b, X=x, Y=y):
    return float(np.mean(np.abs(Y - (a + b * X))))

df[['tv_spend', 'sales']].head()

The gaps a cost is built from

b_sq, a_sq = np.polyfit(x, y, 1)
record('sq_intercept', round(a_sq, 3))
record('sq_slope', round(b_sq, 5))
record('sq_mse', round(mse(a_sq, b_sq), 3))
record('sq_mae', round(mae(a_sq, b_sq), 3))

resid = y - (a_sq + b_sq * x)
record('biggest_gap', round(float(np.max(np.abs(resid))), 2))
record('typical_gap', round(float(np.median(np.abs(resid))), 2))

def plot_gaps(ax):
    order = np.argsort(x)
    ax.scatter(x, y, s=14, color='#9aa0aa', zorder=3)
    for xi, yi in zip(x, y):
        ax.plot([xi, xi], [yi, a_sq + b_sq * xi], color='#e2574c', lw=0.7, alpha=0.55, zorder=2)
    ax.plot(x[order], (a_sq + b_sq * x)[order], color='#3b6fd4', lw=2, zorder=4)
    ax.set_xlabel('TV budget ($ thousands)')
    ax.set_ylabel('Sales (thousands of units)')

save_fig('the-gaps', plot_gaps)

Fit the same straight line under each cost

best_abs = minimize(lambda p: mae(p[0], p[1]), [a_sq, b_sq], method='Nelder-Mead',
                    options=dict(xatol=1e-9, fatol=1e-11, maxiter=50000))
a_ab, b_ab = best_abs.x
record('abs_intercept', round(a_ab, 3))
record('abs_slope', round(b_ab, 5))
record('abs_mse', round(mse(a_ab, b_ab), 3))
record('abs_mae', round(mae(a_ab, b_ab), 3))

# The cross-table the lesson prints: each line scored under BOTH costs.
record('cross_columns', ['', 'Squared error', 'Absolute error'])
record('cross_rows', [
    [f'The line squared error picks (slope {b_sq:.4f})', f'{mse(a_sq, b_sq):.3f}', f'{mae(a_sq, b_sq):.3f}'],
    [f'The line absolute error picks (slope {b_ab:.4f})', f'{mse(a_ab, b_ab):.3f}', f'{mae(a_ab, b_ab):.3f}'],
])
print(f'squared  picks slope {b_sq:.5f}, intercept {a_sq:.3f}')
print(f'absolute picks slope {b_ab:.5f}, intercept {a_ab:.3f}')

Two costs, two lines

def plot_two(ax):
    grid = np.linspace(x.min(), x.max(), 200)
    ax.scatter(x, y, s=14, color='#c9c9d1', zorder=2)
    ax.plot(grid, a_sq + b_sq * grid, color='#e2574c', lw=2.2,
            label=f'squared error  (slope {b_sq:.4f})', zorder=4)
    ax.plot(grid, a_ab + b_ab * grid, color='#3b6fd4', lw=2.2, ls='--',
            label=f'absolute error  (slope {b_ab:.4f})', zorder=3)
    ax.set_xlabel('TV budget ($ thousands)')
    ax.set_ylabel('Sales (thousands of units)')
    ax.legend(frameon=False, loc='upper left')

save_fig('two-costs-two-lines', plot_two)

What one odd market does to each line

OUT_TV, OUT_SALES = 10.0, 26.0
record('outlier_tv', OUT_TV)
record('outlier_sales', OUT_SALES)

X2, Y2 = np.append(x, OUT_TV), np.append(y, OUT_SALES)
b_sq2, a_sq2 = np.polyfit(X2, Y2, 1)
r2 = minimize(lambda p: mae(p[0], p[1], X2, Y2), [a_ab, b_ab], method='Nelder-Mead',
              options=dict(xatol=1e-9, fatol=1e-11, maxiter=50000))
a_ab2, b_ab2 = r2.x

moved_sq, moved_ab = abs(b_sq2 - b_sq), abs(b_ab2 - b_ab)
record('sq_slope_after', round(b_sq2, 5))
record('abs_slope_after', round(b_ab2, 5))
record('sq_moved', f'{moved_sq:.5f}')
record('abs_moved', f'{moved_ab:.5f}')
record('moved_ratio', round(moved_sq / moved_ab))
record('outlier_gap', round(float(abs(OUT_SALES - (a_sq + b_sq * OUT_TV))), 1))
record('outlier_gap_squared', round(float((OUT_SALES - (a_sq + b_sq * OUT_TV)) ** 2)))

record('outlier_columns', ['', 'Slope before', 'Slope after', 'Moved by'])
record('outlier_rows', [
    ['Squared error', f'{b_sq:.4f}', f'{b_sq2:.4f}', f'{moved_sq:.4f}'],
    ['Absolute error', f'{b_ab:.4f}', f'{b_ab2:.4f}', f'{moved_ab:.4f}'],
])
print(f'squared moved {moved_sq:.5f}, absolute moved {moved_ab:.5f} — {moved_sq/moved_ab:.1f}x')

The same market, added to both

def plot_outlier(ax):
    grid = np.linspace(0, x.max(), 200)
    ax.scatter(x, y, s=13, color='#d5d5db', zorder=2)
    ax.scatter([OUT_TV], [OUT_SALES], s=70, color='#c2871a', zorder=6, marker='D')
    ax.annotate('the added market', (OUT_TV, OUT_SALES), textcoords='offset points',
                xytext=(12, -2), fontsize=9, color='#77777f')
    ax.plot(grid, a_sq + b_sq * grid, color='#e2574c', lw=1.2, alpha=0.45, zorder=3)
    ax.plot(grid, a_sq2 + b_sq2 * grid, color='#e2574c', lw=2.2, zorder=5,
            label=f'squared error, moved {moved_sq:.4f}')
    ax.plot(grid, a_ab + b_ab * grid, color='#3b6fd4', lw=1.2, alpha=0.45, ls='--', zorder=3)
    ax.plot(grid, a_ab2 + b_ab2 * grid, color='#3b6fd4', lw=2.2, ls='--', zorder=4,
            label=f'absolute error, moved {moved_ab:.4f}')
    ax.set_xlabel('TV budget ($ thousands)')
    ax.set_ylabel('Sales (thousands of units)')
    ax.legend(frameon=False, loc='lower right')

save_fig('what-one-outlier-does', plot_outlier)

Try this in Expedify →

Related lessons

12 min read

Base rates — what a piece of evidence is actually worth

A face-recognition system that is 99.9% accurate and almost entirely wrong, and a number that sent an innocent woman to prison. Both are the same arithmetic, and it is the arithmetic that decides what any piece of evidence is worth.

Read

10 min read

Confirmation and survivorship — what you never looked for

Two questions about evidence you did not go looking for. One is a rule you have to discover, and one is a pattern in five famous people — and in both, the thing that would have told you the truth is the thing nobody checks.

Read

14 min read

Loss aversion, sunk cost and regression — what it costs you

Four questions you answer about yourself rather than about a scenario, and your own answers are the finding. Then the pattern that makes praise look useless and criticism look like it works, whatever you actually do.

Read