{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2867cb12",
   "metadata": {},
   "source": [
    "# What learning is · Only hit or miss\n",
    "\n",
    "The coach is still standing at the target. They are now allowed **one word**.\n",
    "\n",
    "*Hit.* Or *miss.* Not how far off, not which side — one word, and the robot has\n",
    "to get to the same place on it.\n",
    "\n",
    "This is the third and last thing the robot can be told, and it is a different\n",
    "kind of problem from the other two. \"Eleven left\" says where to go. \"Left\" at\n",
    "least says which way. **\"Miss\" says nothing about direction at all**, so the\n",
    "robot cannot correct. It can only try something and find out whether it did\n",
    "better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e501f99c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-19T08:41:05.332569Z",
     "iopub.status.busy": "2026-08-19T08:41:05.332130Z",
     "iopub.status.idle": "2026-08-19T08:41:05.711251Z",
     "shell.execute_reply": "2026-08-19T08:41:05.710909Z"
    }
   },
   "outputs": [],
   "source": [
    "# hide — plumbing so this file runs both in the repo and in Colab.\n",
    "try:\n",
    "    from _figkit import save_fig, record\n",
    "except ImportError:  # Colab — no repo, no problem\n",
    "    def save_fig(name, plot, **kw):\n",
    "        import matplotlib.pyplot as plt\n",
    "        fig, ax = plt.subplots(figsize=kw.get('figsize', (7, 4.2))); plot(ax); plt.show()\n",
    "    def record(key, value):\n",
    "        print(f'{key} = {value}'); return value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd3dbe52",
   "metadata": {},
   "source": [
    "## 1. One measuring stick for all three\n",
    "\n",
    "The earlier lessons each measured their own learner, and comparing numbers\n",
    "across notebooks is how people fool themselves. So all three are re-run here,\n",
    "scored identically:\n",
    "\n",
    "> **how many arrows before the aim is within 2.5 cm of the setting that exactly\n",
    "> cancels the habit.**\n",
    "\n",
    "That is a fact about what the robot has *learned* rather than about how lucky\n",
    "its last few shots were, which matters here because the third learner spends a\n",
    "lot of arrows deliberately shooting badly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "534e4541",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-19T08:41:05.713208Z",
     "iopub.status.busy": "2026-08-19T08:41:05.713051Z",
     "iopub.status.idle": "2026-08-19T08:41:05.718184Z",
     "shell.execute_reply": "2026-08-19T08:41:05.717861Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "near_cm = 2.5\n",
      "gold_cm = 10\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| caption: The same bow, the same target, and one scoring rule for all three\n",
    "import random\n",
    "\n",
    "BIAS = (-11.0, 7.0)\n",
    "WOBBLE = 4.0\n",
    "STEP = 0.15        # told a number\n",
    "CAT_STEP = 0.6     # told a category\n",
    "GOLD = 10.0        # inside this ring, the coach says 'hit'\n",
    "NEAR = 2.5         # 'has learned it': aim this close to perfect\n",
    "BUDGET = 4000      # arrows after which we stop waiting\n",
    "\n",
    "IDEAL = (-BIAS[0], -BIAS[1])\n",
    "\n",
    "def residual(aim):\n",
    "    return ((aim[0] - IDEAL[0]) ** 2 + (aim[1] - IDEAL[1]) ** 2) ** 0.5\n",
    "\n",
    "def sign(v):\n",
    "    return 1.0 if v > 0 else (-1.0 if v < 0 else 0.0)\n",
    "\n",
    "def fire(aim, rng):\n",
    "    return (aim[0] + BIAS[0] + rng.gauss(0, WOBBLE),\n",
    "            aim[1] + BIAS[1] + rng.gauss(0, WOBBLE))\n",
    "\n",
    "def told(seed, kind, budget=400):\n",
    "    \"\"\"Told a number, or told a category. Both can correct; only the step differs.\"\"\"\n",
    "    rng = random.Random(seed)\n",
    "    aim = [0.0, 0.0]\n",
    "    for n in range(1, budget + 1):\n",
    "        shot = fire(aim, rng)\n",
    "        if kind == 'number':\n",
    "            aim[0] -= STEP * shot[0]\n",
    "            aim[1] -= STEP * shot[1]\n",
    "        else:\n",
    "            aim[0] -= CAT_STEP * sign(shot[0])\n",
    "            aim[1] -= CAT_STEP * sign(shot[1])\n",
    "        if residual(aim) < NEAR:\n",
    "            return n\n",
    "    return budget\n",
    "\n",
    "record('near_cm', NEAR)\n",
    "record('gold_cm', int(GOLD))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65a70ca9",
   "metadata": {},
   "source": [
    "## 2. Learning from one word\n",
    "\n",
    "It cannot correct, so it does the only thing left: **try a different setting and\n",
    "see whether it hits more often.**\n",
    "\n",
    "Ten arrows at where it currently aims, ten at a nearby guess, keep whichever\n",
    "scored better, repeat. Every one of those arrows is spent, including the ones\n",
    "fired deliberately in the wrong place — that is what exploring costs.\n",
    "\n",
    "Both settings are re-measured every round, on purpose. Measuring the incumbent\n",
    "once and remembering the score sounds like a saving and is a trap: a lucky ten\n",
    "out of ten can never be beaten, and the robot freezes for good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ecd7aa45",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-19T08:41:05.719548Z",
     "iopub.status.busy": "2026-08-19T08:41:05.719460Z",
     "iopub.status.idle": "2026-08-19T08:41:05.798237Z",
     "shell.execute_reply": "2026-08-19T08:41:05.797943Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     told a number: 12 arrows\n",
      "   told a category: 21 arrows\n",
      "  told hit or miss: 517 arrows\n",
      "arrows_number = 12\n",
      "arrows_category = 21\n",
      "arrows_hitmiss = 517\n",
      "times_more = 43\n",
      "figure what-one-word-costs -> only-hit-or-miss.what-one-word-costs.{light,dark}.svg\n"
     ]
    }
   ],
   "source": [
    "#| caption: Try something, see if it hits more often, keep it if it did\n",
    "BATCH = 10         # arrows spent judging one setting\n",
    "SIGMA = 4.0        # how far a guess strays from where it is now\n",
    "\n",
    "def hit_or_miss(seed, gold=GOLD, budget=BUDGET):\n",
    "    rng = random.Random(seed)\n",
    "    aim = [0.0, 0.0]\n",
    "    spent = 0\n",
    "\n",
    "    def hit_rate(a):\n",
    "        \"\"\"Fire BATCH arrows at `a`. All the coach ever says is hit or miss.\"\"\"\n",
    "        nonlocal spent\n",
    "        hits = 0\n",
    "        for _ in range(BATCH):\n",
    "            shot = fire(a, rng)\n",
    "            spent += 1\n",
    "            hits += 1 if (shot[0] ** 2 + shot[1] ** 2) ** 0.5 <= gold else 0\n",
    "        return hits / BATCH\n",
    "\n",
    "    while spent < budget:\n",
    "        here = hit_rate(aim)                                  # re-measured, every round\n",
    "        guess = [aim[0] + rng.gauss(0, SIGMA), aim[1] + rng.gauss(0, SIGMA)]\n",
    "        there = hit_rate(guess)\n",
    "        if there > here:\n",
    "            aim = guess\n",
    "        if residual(aim) < NEAR:\n",
    "            return spent\n",
    "    return budget\n",
    "\n",
    "def average(f, trials=40, **kw):\n",
    "    return round(sum(f(s, **kw) for s in range(trials)) / trials)\n",
    "\n",
    "arrows = {\n",
    "    'told a number': average(lambda s: told(s, 'number')),\n",
    "    'told a category': average(lambda s: told(s, 'category')),\n",
    "    'told hit or miss': average(hit_or_miss),\n",
    "}\n",
    "for label, n in arrows.items():\n",
    "    print(f'{label:>18}: {n} arrows')\n",
    "\n",
    "record('arrows_number', arrows['told a number'])\n",
    "record('arrows_category', arrows['told a category'])\n",
    "record('arrows_hitmiss', arrows['told hit or miss'])\n",
    "record('times_more', round(arrows['told hit or miss'] / arrows['told a number']))\n",
    "\n",
    "def plot_cost(ax):\n",
    "    labels = list(arrows)\n",
    "    values = list(arrows.values())\n",
    "    bars = ax.bar(labels, values, color=['#9aa1ab', '#3b6fd4', '#ee785b'])\n",
    "    for b, v in zip(bars, values):\n",
    "        ax.text(b.get_x() + b.get_width() / 2, v + max(values) * 0.02, f'{v}',\n",
    "                ha='center', fontsize=11)\n",
    "    ax.set_ylabel('arrows until it has learned the habit')\n",
    "    ax.set_ylim(0, max(values) * 1.16)\n",
    "    ax.spines[['top', 'right']].set_visible(False)\n",
    "\n",
    "save_fig('what-one-word-costs', plot_cost, figsize=(7.0, 3.8))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f85f92",
   "metadata": {},
   "source": [
    "## 3. When the word stops carrying anything\n",
    "\n",
    "One word is only informative if it sometimes changes.\n",
    "\n",
    "Make the gold ring enormous and the robot hits from anywhere — every arrow comes\n",
    "back *hit*, and a word that is always the same is not feedback. Make it tiny and\n",
    "almost nothing lands inside — every arrow comes back *miss*, and the robot has\n",
    "nothing to steer by either.\n",
    "\n",
    "Somewhere in between the word actually varies with what the robot did, and that\n",
    "is the only place it can learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "52898510",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-19T08:41:05.799806Z",
     "iopub.status.busy": "2026-08-19T08:41:05.799701Z",
     "iopub.status.idle": "2026-08-19T08:41:05.853119Z",
     "shell.execute_reply": "2026-08-19T08:41:05.852854Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hits_at_20 = 60\n",
      "hits_at_6 = 44\n",
      "hits_at_2 = 7\n",
      "hits_at_20 = 60\n",
      "hits_at_6 = 44\n",
      "hits_at_2 = 7\n",
      "figure what-counts-as-a-hit -> only-hit-or-miss.what-counts-as-a-hit.{light,dark}.svg\n"
     ]
    }
   ],
   "source": [
    "#| caption: What counts as a hit, at three sizes of gold ring\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as patches\n",
    "\n",
    "def plot_rings(ax):\n",
    "    rng2 = random.Random(11)\n",
    "    shots = [fire((11.0, -7.0), rng2) for _ in range(60)]      # a robot already aiming well\n",
    "    ax.scatter([s[0] for s in shots], [s[1] for s in shots], s=22, color='#ee785b',\n",
    "               alpha=0.7, zorder=1)\n",
    "    for r, style in ((20, ':'), (6, '--'), (2, '-')):\n",
    "        inside = sum(1 for s in shots if (s[0] ** 2 + s[1] ** 2) ** 0.5 <= r)\n",
    "        ax.add_artist(patches.Circle((0, 0), r, facecolor='none', edgecolor='#3a3a3a',\n",
    "                                     lw=1.4, ls=style, zorder=2))\n",
    "        ax.annotate(f'{r} cm ring — {inside} of {len(shots)} arrows are a \"hit\"',\n",
    "                    xy=(r * 0.71, r * 0.71), xytext=(26, r - 1), fontsize=9.5, color='#3a3a3a',\n",
    "                    va='center', arrowprops=dict(arrowstyle='-', color='#b9b2aa', lw=1))\n",
    "        record(f'hits_at_{r}', inside)\n",
    "    ax.set_xlim(-24, 68); ax.set_ylim(-24, 24); ax.set_aspect('equal')\n",
    "    ax.set_xticks([]); ax.set_yticks([]); ax.grid(False)\n",
    "    for side in ('top', 'right', 'bottom', 'left'):\n",
    "        ax.spines[side].set_visible(False)\n",
    "\n",
    "save_fig('what-counts-as-a-hit', plot_rings, figsize=(7.4, 3.4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "42671947",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-08-19T08:41:05.854361Z",
     "iopub.status.busy": "2026-08-19T08:41:05.854287Z",
     "iopub.status.idle": "2026-08-19T08:41:06.038980Z",
     "shell.execute_reply": "2026-08-19T08:41:06.038737Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gold  2 cm ->  1156 arrows\n",
      "gold  3 cm ->   753 arrows\n",
      "gold  6 cm ->   471 arrows\n",
      "gold 10 cm ->   475 arrows\n",
      "gold 14 cm ->  1528 arrows\n",
      "gold 20 cm ->  3969 arrows  (never got there — stopped at the budget)\n",
      "budget = 4000\n",
      "cost_tiny = 1156\n",
      "cost_best = 471\n",
      "best_gold = 6\n",
      "cost_huge = 3969\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "figure a-word-that-never-changes -> only-hit-or-miss.a-word-that-never-changes.{light,dark}.svg\n"
     ]
    }
   ],
   "source": [
    "#| caption: The cost of one word, against how often that word changes\n",
    "golds = [2, 3, 6, 10, 14, 20]\n",
    "costs = [average(hit_or_miss, trials=30, gold=float(g)) for g in golds]\n",
    "for g, c in zip(golds, costs):\n",
    "    cap = '  (never got there — stopped at the budget)' if c >= BUDGET * 0.95 else ''\n",
    "    print(f'gold {g:>2} cm -> {c:>5} arrows{cap}')\n",
    "\n",
    "record('budget', BUDGET)\n",
    "record('cost_tiny', costs[0])\n",
    "record('cost_best', min(costs))\n",
    "record('best_gold', golds[costs.index(min(costs))])\n",
    "record('cost_huge', costs[-1])\n",
    "\n",
    "def plot_signal(ax):\n",
    "    ax.plot(golds, costs, color='#ee785b', linewidth=2.4, marker='o', markersize=7)\n",
    "    ax.axhline(BUDGET, color='#6b7280', linestyle=':', linewidth=1.2)\n",
    "    ax.text(3, BUDGET * 0.93, 'gave up here', fontsize=9.5, color='#6b7280')\n",
    "    ax.annotate('almost never hits\\nevery word is \"miss\"', xy=(2, costs[0]),\n",
    "                xytext=(2.6, costs[0] + BUDGET * 0.16), fontsize=9.5, color='#6b7280')\n",
    "    ax.annotate('almost always hits\\nevery word is \"hit\"', xy=(20, costs[-1]),\n",
    "                xytext=(12.4, costs[-1] - BUDGET * 0.20), fontsize=9.5, color='#6b7280')\n",
    "    ax.set_xlabel('how big the gold ring is (cm)')\n",
    "    ax.set_ylabel('arrows until it has learned the habit')\n",
    "    ax.set_ylim(0, BUDGET * 1.1)\n",
    "    ax.spines[['top', 'right']].set_visible(False)\n",
    "\n",
    "save_fig('a-word-that-never-changes', plot_signal, figsize=(7.2, 4.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad6854c",
   "metadata": {},
   "source": [
    "## 4. So why would anyone accept this\n",
    "\n",
    "Because for a great many problems, one word is the only feedback that exists.\n",
    "\n",
    "Nobody can tell a game-playing program the correct move — only who won. Nobody\n",
    "can tell a system the correct thing to have said — only whether the person came\n",
    "back. The exact miss, when you can get it, is worth roughly forty arrows for\n",
    "every one it costs. The point of this method is the problems where nobody can\n",
    "give it to you at any price."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}