{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f54b8e5",
   "metadata": {},
   "source": [
    "# Introduction to AI · Where judgment fails\n",
    "\n",
    "Most of this module's fifteen questions are **split** — two arms of the same\n",
    "decision, differing in one word — so their reveal is what the other arm was\n",
    "asked and needs no arithmetic at all.\n",
    "\n",
    "Three things do need computing, and they are the three that carry the argument:\n",
    "\n",
    "1. **A flag on a rare event is mostly false.** Counted, not simulated.\n",
    "2. **Extreme results move back to the middle on their own**, with nobody praised\n",
    "   and nobody scolded.\n",
    "3. **Half of an expert's error is inconsistency rather than bias** — which is the\n",
    "   claim the whole module turns on.\n",
    "\n",
    "Every figure and number this notebook produces is what the lessons print.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "abe7cf65",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-25T02:36:36.022706Z",
     "iopub.status.busy": "2026-08-25T02:36:36.022523Z",
     "iopub.status.idle": "2026-08-25T02:36:36.314132Z",
     "shell.execute_reply": "2026-08-25T02:36:36.313423Z"
    }
   },
   "outputs": [],
   "source": [
    "# hide — plumbing so this file runs both in the repo and in Colab.\n",
    "try:\n",
    "    from _figkit import save_fig, record\n",
    "except ImportError:  # Colab — no repo, no problem\n",
    "    def save_fig(name, plot, **kw):\n",
    "        import matplotlib.pyplot as plt\n",
    "        fig, ax = plt.subplots(figsize=kw.get('figsize', (7, 4.2))); plot(ax); plt.show()\n",
    "    def record(key, value):\n",
    "        print(f'{key} = {value}'); return value\n",
    "\n",
    "import numpy as np\n",
    "rng = np.random.default_rng(11)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ceed539",
   "metadata": {},
   "source": [
    "## 1. A face-recognition system at an airport\n",
    "\n",
    "The system is 99.9% accurate in both directions. It is looking for people who\n",
    "are about one in ten million. Fifty million passengers pass through in a year.\n",
    "\n",
    "Nothing here is a flaw in the system. It is a very good system.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1491a34e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-25T02:36:36.317212Z",
     "iopub.status.busy": "2026-08-25T02:36:36.316303Z",
     "iopub.status.idle": "2026-08-25T02:36:36.325143Z",
     "shell.execute_reply": "2026-08-25T02:36:36.324858Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "air_passengers = 50,000,000\n",
      "air_wanted = 5\n",
      "air_accuracy = 99.9%\n",
      "air_caught = 5\n",
      "air_false = 50,000\n",
      "air_alarms = 50,005\n",
      "air_worth = 1 in 10,001\n",
      "air_per_day = 137\n",
      "air_columns = ['', 'System raises an alarm', 'It does not', 'All']\n",
      "air_rows = [['Actually wanted', '5', '0', '5'], ['An ordinary passenger', '50,000', '49,949,995', '49,999,995'], ['All', '50,005', '49,949,995', '50,000,000']]\n",
      "5 real of 50,005 alarms — about 1 in 10,001\n",
      "137 innocent people stopped every day\n"
     ]
    }
   ],
   "source": [
    "#| caption: Fifty million passengers, and where the alarms land\n",
    "PASSENGERS, RATE, ACC = 50_000_000, 1 / 10_000_000, 0.999\n",
    "wanted = round(PASSENGERS * RATE)\n",
    "ordinary = PASSENGERS - wanted\n",
    "caught = round(wanted * ACC)\n",
    "false_alarms = round(ordinary * (1 - ACC))\n",
    "alarms = caught + false_alarms\n",
    "\n",
    "record('air_passengers', f'{PASSENGERS:,}')\n",
    "record('air_wanted', wanted)\n",
    "record('air_accuracy', f'{ACC:.1%}')\n",
    "record('air_caught', caught)\n",
    "record('air_false', f'{false_alarms:,}')\n",
    "record('air_alarms', f'{alarms:,}')\n",
    "record('air_worth', f'1 in {round(alarms / caught):,}')\n",
    "record('air_per_day', f'{round(false_alarms / 365):,}')\n",
    "record('air_columns', ['', 'System raises an alarm', 'It does not', 'All'])\n",
    "record('air_rows', [\n",
    "    ['Actually wanted', f'{caught}', f'{wanted - caught}', f'{wanted}'],\n",
    "    ['An ordinary passenger', f'{false_alarms:,}', f'{ordinary - false_alarms:,}', f'{ordinary:,}'],\n",
    "    ['All', f'{alarms:,}', f'{PASSENGERS - alarms:,}', f'{PASSENGERS:,}'],\n",
    "])\n",
    "print(f'{caught} real of {alarms:,} alarms — about 1 in {round(alarms/caught):,}')\n",
    "print(f'{round(false_alarms/365):,} innocent people stopped every day')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f930d95f",
   "metadata": {},
   "source": [
    "## 2. The number that convicted Sally Clark\n",
    "\n",
    "An expert witness told the court that the chance of two cot deaths in one\n",
    "family was 1 in 73 million. The figure came from taking a single cot death's\n",
    "probability and squaring it.\n",
    "\n",
    "Squaring assumes the two deaths are independent — that the second is no more\n",
    "likely after the first. Whatever raises the risk for one baby, from genes to a\n",
    "household, is still there for the second.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "74d8b899",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-25T02:36:36.326277Z",
     "iopub.status.busy": "2026-08-25T02:36:36.326187Z",
     "iopub.status.idle": "2026-08-25T02:36:36.328816Z",
     "shell.execute_reply": "2026-08-25T02:36:36.328635Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clark_single = 1 in 8,500\n",
      "clark_squared = 1 in 72,250,000\n",
      "clark_columns = ['If a first death makes a second…', 'Chance of two']\n",
      "clark_rows = [['no more likely (what the court was told)', '1 in 72,250,000'], ['5 times more likely', '1 in 14,450,000'], ['10 times more likely', '1 in 7,225,000']]\n",
      "  1x -> 1 in 72,250,000\n",
      "  5x -> 1 in 14,450,000\n",
      " 10x -> 1 in 7,225,000\n"
     ]
    }
   ],
   "source": [
    "#| caption: What squaring assumes, and what happens if it is wrong\n",
    "SINGLE = 1 / 8_500          # roughly the rate the court was given\n",
    "record('clark_single', '1 in 8,500')\n",
    "record('clark_squared', f'1 in {round(1 / SINGLE**2):,}')\n",
    "\n",
    "# If a first cot death makes a second more likely — as later evidence\n",
    "# suggested — the honest figure moves by orders of magnitude.\n",
    "record('clark_columns', ['If a first death makes a second…', 'Chance of two'])\n",
    "record('clark_rows', [\n",
    "    ['no more likely (what the court was told)', f'1 in {round(1 / SINGLE**2):,}'],\n",
    "    ['5 times more likely', f'1 in {round(1 / (SINGLE * SINGLE * 5)):,}'],\n",
    "    ['10 times more likely', f'1 in {round(1 / (SINGLE * SINGLE * 10)):,}'],\n",
    "])\n",
    "for mult in (1, 5, 10):\n",
    "    print(f'{mult:>3}x -> 1 in {round(1 / (SINGLE * SINGLE * mult)):,}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bae55cb5",
   "metadata": {},
   "source": [
    "## 5. Nobody was praised and nobody was scolded\n",
    "\n",
    "Two hundred people. Their true skill never changes. Each quarter's result is\n",
    "that skill plus luck. Take the best 10% of quarter one and follow them.\n",
    "\n",
    "**No coaching happens in this simulation at all.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d2801d35",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-25T02:36:36.329791Z",
     "iopub.status.busy": "2026-08-25T02:36:36.329721Z",
     "iopub.status.idle": "2026-08-25T02:36:36.397905Z",
     "shell.execute_reply": "2026-08-25T02:36:36.397684Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rep_n = 200\n",
      "top_q1 = 139\n",
      "top_q2 = 114\n",
      "top_got_worse = 90%\n",
      "bot_q1 = 62\n",
      "bot_q2 = 86\n",
      "bot_improved = 80%\n",
      "figure nobody-was-praised -> where-judgment-fails.nobody-was-praised.{light,dark}.svg\n",
      "best 10%: 139 -> 114, 90% got worse\n",
      "worst 10%: 62 -> 86, 80% improved\n"
     ]
    }
   ],
   "source": [
    "#| caption: The best and the worst, followed into a second quarter\n",
    "n = 200\n",
    "skill = rng.normal(100, 12, n)\n",
    "q1 = skill + rng.normal(0, 18, n)\n",
    "q2 = skill + rng.normal(0, 18, n)\n",
    "\n",
    "top, bot = np.argsort(q1)[-20:], np.argsort(q1)[:20]\n",
    "record('rep_n', n)\n",
    "record('top_q1', f'{q1[top].mean():.0f}')\n",
    "record('top_q2', f'{q2[top].mean():.0f}')\n",
    "record('top_got_worse', f'{(q2[top] < q1[top]).mean():.0%}')\n",
    "record('bot_q1', f'{q1[bot].mean():.0f}')\n",
    "record('bot_q2', f'{q2[bot].mean():.0f}')\n",
    "record('bot_improved', f'{(q2[bot] > q1[bot]).mean():.0%}')\n",
    "\n",
    "def plot_reps(ax):\n",
    "    for grp, colour, label in ((top, '#e2574c', 'best 10% of quarter one'),\n",
    "                               (bot, '#3b6fd4', 'worst 10% of quarter one')):\n",
    "        for i in grp:\n",
    "            ax.plot([0, 1], [q1[i], q2[i]], color=colour, alpha=0.16, lw=1)\n",
    "        ax.plot([0, 1], [q1[grp].mean(), q2[grp].mean()], color=colour, lw=2.6, label=label)\n",
    "    ax.axhline(skill.mean(), color='#9aa0aa', ls='--', lw=1)\n",
    "    ax.annotate('everyone\\'s average', (1, skill.mean()), textcoords='offset points',\n",
    "                xytext=(-6, 6), ha='right', fontsize=9, color='#77777f')\n",
    "    ax.set_xticks([0, 1]); ax.set_xticklabels(['Quarter one', 'Quarter two'])\n",
    "    ax.set_ylabel('% of target')\n",
    "    ax.legend(frameon=False, loc='upper center')\n",
    "    ax.grid(axis='x', visible=False)\n",
    "\n",
    "save_fig('nobody-was-praised', plot_reps, figsize=(6.8, 4.0))\n",
    "print(f'best 10%: {q1[top].mean():.0f} -> {q2[top].mean():.0f}, {(q2[top]<q1[top]).mean():.0%} got worse')\n",
    "print(f'worst 10%: {q1[bot].mean():.0f} -> {q2[bot].mean():.0f}, {(q2[bot]>q1[bot]).mean():.0%} improved')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db96d53c",
   "metadata": {},
   "source": [
    "## 6. Two judges and one rule\n",
    "\n",
    "Five hundred cases with a true value. Two underwriters share the **same habit**\n",
    "— both run 6 points generous — and each has their own wobble on top. The rule\n",
    "has the identical 6-point habit and no wobble at all.\n",
    "\n",
    "If a rule beat them by being fairer, this would not work. It is not fairer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9a51fb5b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-25T02:36:36.399103Z",
     "iopub.status.busy": "2026-08-25T02:36:36.399023Z",
     "iopub.status.idle": "2026-08-25T02:36:36.447705Z",
     "shell.execute_reply": "2026-08-25T02:36:36.447482Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "judge_habit = 6\n",
      "judge_cases = 500\n",
      "judge1_error = 12.2\n",
      "judge2_error = 12.6\n",
      "rule_error = 6.0\n",
      "judges_disagree_by = 12.6\n",
      "wobble_share = 51%\n",
      "judge_columns = ['', 'Habit', 'Wobble', 'Typical error']\n",
      "judge_rows = [['Underwriter one', '+6', 'yes', '12.2'], ['Underwriter two', '+6', 'yes', '12.6'], ['A written-down rule', '+6', 'none', '6.0']]\n",
      "figure habit-and-wobble -> where-judgment-fails.habit-and-wobble.{light,dark}.svg\n",
      "judges 12.2 and 12.6; rule 6.0; they disagree by 12.6\n",
      "51% of the error was wobble\n"
     ]
    }
   ],
   "source": [
    "#| caption: The same habit, with and without a wobble\n",
    "HABIT, WOBBLE, M = 6.0, 11.0, 500\n",
    "truth = rng.normal(50, 15, M)\n",
    "judge1 = truth + HABIT + rng.normal(0, WOBBLE, M)\n",
    "judge2 = truth + HABIT + rng.normal(0, WOBBLE, M)\n",
    "rule = truth + HABIT\n",
    "\n",
    "err = lambda p: float(np.sqrt(np.mean((p - truth) ** 2)))\n",
    "e1, e2, er = err(judge1), err(judge2), err(rule)\n",
    "disagree = float(np.mean(np.abs(judge1 - judge2)))\n",
    "\n",
    "record('judge_habit', f'{HABIT:.0f}')\n",
    "record('judge_cases', M)\n",
    "record('judge1_error', f'{e1:.1f}')\n",
    "record('judge2_error', f'{e2:.1f}')\n",
    "record('rule_error', f'{er:.1f}')\n",
    "record('judges_disagree_by', f'{disagree:.1f}')\n",
    "record('wobble_share', f'{(e1 - er) / e1:.0%}')\n",
    "record('judge_columns', ['', 'Habit', 'Wobble', 'Typical error'])\n",
    "record('judge_rows', [\n",
    "    ['Underwriter one', f'+{HABIT:.0f}', 'yes', f'{e1:.1f}'],\n",
    "    ['Underwriter two', f'+{HABIT:.0f}', 'yes', f'{e2:.1f}'],\n",
    "    ['A written-down rule', f'+{HABIT:.0f}', 'none', f'{er:.1f}'],\n",
    "])\n",
    "\n",
    "def plot_judges(ax):\n",
    "    ax.bar(['Underwriter\\none', 'Underwriter\\ntwo', 'A written-down\\nrule'],\n",
    "           [e1, e2, er], color=['#9aa0aa', '#9aa0aa', '#e2574c'])\n",
    "    for i, v in enumerate([e1, e2, er]):\n",
    "        ax.text(i, v + 0.25, f'{v:.1f}', ha='center', fontsize=11)\n",
    "    ax.axhline(HABIT, color='#3b6fd4', ls='--', lw=1.2)\n",
    "    ax.annotate('the habit they all share', (2.4, HABIT), textcoords='offset points',\n",
    "                xytext=(0, 6), ha='right', fontsize=9, color='#3b6fd4')\n",
    "    ax.set_ylabel('Typical error')\n",
    "    ax.grid(axis='x', visible=False)\n",
    "\n",
    "save_fig('habit-and-wobble', plot_judges, figsize=(6.6, 3.8))\n",
    "print(f'judges {e1:.1f} and {e2:.1f}; rule {er:.1f}; they disagree by {disagree:.1f}')\n",
    "print(f'{(e1-er)/e1:.0%} of the error was wobble')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e904736",
   "metadata": {},
   "source": [
    "## What this notebook establishes\n",
    "\n",
    "- A flag on a rare event is mostly false: **45 of 140** flagged customers leave,\n",
    "  against the 90% the model's catch rate suggests.\n",
    "- Extreme results move back towards the middle **with no coaching at all**.\n",
    "- Two experts with an **identical** habit still differ from each other, and a\n",
    "  rule that keeps that habit and drops the wobble **halves the error**.\n",
    "\n",
    "**Worth changing:** give the two underwriters *different* habits and watch what\n",
    "the rule can and cannot fix. It removes the wobble either way. It has never\n",
    "touched the habit.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}