{ "cells": [ { "cell_type": "markdown", "id": "c32d5ea4", "metadata": {}, "source": [ "# Decision Trees · Perfect on what it has seen\n", "\n", "The last lesson grew a tree that got 23 of the 24 films right. But it was\n", "scored on the same 24 films it was built from.\n", "\n", "That score cannot tell you much. A tree can store an answer for every row it\n", "was given. It will then be right about all of them and know nothing about\n", "anybody else.\n", "\n", "There is a simple test. **Keep some of the data back. Build the tree without\n", "it. Then ask the tree about it.**\n", "\n", "This notebook runs that test on 1,500 real employees, at every depth from 1 to\n", "20." ] }, { "cell_type": "code", "execution_count": 1, "id": "1ea1a360", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:27.293852Z", "iopub.status.busy": "2026-08-20T05:48:27.293672Z", "iopub.status.idle": "2026-08-20T05:48:27.955249Z", "shell.execute_reply": "2026-08-20T05:48:27.954985Z" } }, "outputs": [], "source": [ "# hide — plumbing so this file runs both in the repo and in Colab.\n", "try:\n", " from _figkit import save_fig, record\n", "except ImportError: # Colab — no repo, no problem\n", " def save_fig(name, plot, **kw):\n", " import matplotlib.pyplot as plt\n", " fig, ax = plt.subplots(figsize=kw.get('figsize', (7, 4.2))); plot(ax); plt.show()\n", " def record(key, value):\n", " print(f'{key} = {value}'); return value" ] }, { "cell_type": "markdown", "id": "4ef4a635", "metadata": {}, "source": [ "## 1. The data\n", "\n", "1,500 people. For each one we know 22 things: age, income, how far they live\n", "from work, how long they have been in the role, whether they work overtime, and\n", "so on. One more column says whether they left the company.\n", "\n", "The employee ID is left out. It is a number the company assigned. It says\n", "nothing about the person. The next lesson puts it back and shows what happens." ] }, { "cell_type": "code", "execution_count": 2, "id": "d24bc7e2", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:27.956576Z", "iopub.status.busy": "2026-08-20T05:48:27.956469Z", "iopub.status.idle": "2026-08-20T05:48:29.597963Z", "shell.execute_reply": "2026-08-20T05:48:29.597746Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "n_people = 1500\n", "n_columns = 22\n", "n_left = 202\n", "share_left = 0.135\n", "share_left_pct = 13.5%\n", "n_stayed = 1298\n", "1500 people, 22 things known about each, 202 of them left\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EmployeeIDAgeGenderDepartmentEducationJobLevelMonthlyIncomeDistanceFromHomeNumCompaniesWorkedTotalWorkingYears...TrainingTimesLastYearPercentSalaryHikeStockOptionLevelOverTimeEnvironmentSatisfactionJobSatisfactionWorkLifeBalanceRelationshipSatisfactionPerformanceRatingAttrition
0100156MaleR&D21855112438...3183Yes34423No
1100246FemaleHR33146622427...5221No33434No
2100332MaleSales34143351111...6130Yes44223Yes
3100425MaleR&D2313900703...3221No31333No
4100538MaleHR12126816317...2161No22443No
\n", "

5 rows × 24 columns

\n", "
" ], "text/plain": [ " EmployeeID Age Gender Department Education JobLevel MonthlyIncome \\\n", "0 1001 56 Male R&D 2 1 8551 \n", "1 1002 46 Female HR 3 3 14662 \n", "2 1003 32 Male Sales 3 4 14335 \n", "3 1004 25 Male R&D 2 3 13900 \n", "4 1005 38 Male HR 1 2 12681 \n", "\n", " DistanceFromHome NumCompaniesWorked TotalWorkingYears ... \\\n", "0 12 4 38 ... \n", "1 2 4 27 ... \n", "2 1 1 11 ... \n", "3 7 0 3 ... \n", "4 6 3 17 ... \n", "\n", " TrainingTimesLastYear PercentSalaryHike StockOptionLevel OverTime \\\n", "0 3 18 3 Yes \n", "1 5 22 1 No \n", "2 6 13 0 Yes \n", "3 3 22 1 No \n", "4 2 16 1 No \n", "\n", " EnvironmentSatisfaction JobSatisfaction WorkLifeBalance \\\n", "0 3 4 4 \n", "1 3 3 4 \n", "2 4 4 2 \n", "3 3 1 3 \n", "4 2 2 4 \n", "\n", " RelationshipSatisfaction PerformanceRating Attrition \n", "0 2 3 No \n", "1 3 4 No \n", "2 2 3 Yes \n", "3 3 3 No \n", "4 4 3 No \n", "\n", "[5 rows x 24 columns]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| caption: 1,500 employees, and the column we are trying to predict\n", "from pathlib import Path\n", "import pandas as pd\n", "\n", "# Local checkout first, the published copy otherwise — so this cell works\n", "# unchanged in Colab, where there is no repo.\n", "REL = 'public/datasets/decision-trees/hr-employee-attrition.csv'\n", "here = Path.cwd()\n", "LOCAL = next((p / REL for p in [here, *here.parents] if (p / REL).exists()), None)\n", "CSV = LOCAL or 'https://raw.githubusercontent.com/Dr-Shashank-S-Sharma/expedify-ai-courses/main/datasets/decision-trees/hr-employee-attrition.csv'\n", "\n", "df = pd.read_csv(CSV)\n", "record(\"n_people\", len(df))\n", "record(\"n_columns\", df.shape[1] - 2) # not the ID, not the answer\n", "record(\"n_left\", int((df.Attrition == \"Yes\").sum()))\n", "record(\"share_left\", round(float((df.Attrition == \"Yes\").mean()), 3))\n", "record(\"share_left_pct\", f\"{(df.Attrition == 'Yes').mean():.1%}\")\n", "record(\"n_stayed\", int((df.Attrition == \"No\").sum()))\n", "print(f\"{len(df)} people, {df.shape[1] - 2} things known about each, \"\n", " f\"{(df.Attrition == 'Yes').sum()} of them left\")\n", "df.head()" ] }, { "cell_type": "markdown", "id": "087931f4", "metadata": {}, "source": [ "## 2. The number every score has to beat\n", "\n", "Start by asking what you get for free.\n", "\n", "Say \"this person stays\" about everybody. Do not look at a single column. You\n", "will be right about 86.5% of them, because 86.5% of them stayed.\n", "\n", "That number is called the **baseline**. Any model that scores below it has\n", "earned nothing, and a model that only just beats it has earned very little." ] }, { "cell_type": "code", "execution_count": 3, "id": "47750735", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:29.599005Z", "iopub.status.busy": "2026-08-20T05:48:29.598933Z", "iopub.status.idle": "2026-08-20T05:48:29.602801Z", "shell.execute_reply": "2026-08-20T05:48:29.602585Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "baseline = 0.865\n", "baseline_pct = 86.5%\n", "say 'stays' to all 1500 people -> right 86.5% of the time\n", "leavers found by that strategy -> 0 of 202\n" ] } ], "source": [ "#| caption: Say \"stays\" about everyone, and score it\n", "y = (df.Attrition == \"Yes\").astype(int)\n", "X = pd.get_dummies(df.drop(columns=[\"Attrition\", \"EmployeeID\"]), drop_first=True)\n", "\n", "baseline = round(float(1 - y.mean()), 3)\n", "record(\"baseline\", baseline)\n", "# Percentages for the prose. A share of 1.0 interpolates into a sentence as\n", "# \"1\", which reads as one person rather than all of them.\n", "record(\"baseline_pct\", f\"{baseline:.1%}\")\n", "print(f\"say 'stays' to all {len(y)} people -> right {baseline:.1%} of the time\")\n", "print(f\"leavers found by that strategy -> 0 of {int(y.sum())}\")" ] }, { "cell_type": "markdown", "id": "a15eb3eb", "metadata": {}, "source": [ "## 3. Holding data back\n", "\n", "Split the people into two groups.\n", "\n", "The tree is built using the first group only. The second group is kept back and\n", "the tree never sees it. This second group is called the **held-out set** or the\n", "**test set**. Then score the tree on both groups and compare.\n", "\n", "The split keeps the same share of leavers in each group. Otherwise a test set\n", "with too few leavers in it would tell you about the split rather than about the\n", "model." ] }, { "cell_type": "code", "execution_count": 4, "id": "c67e352d", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:29.603765Z", "iopub.status.busy": "2026-08-20T05:48:29.603693Z", "iopub.status.idle": "2026-08-20T05:48:30.791562Z", "shell.execute_reply": "2026-08-20T05:48:30.791274Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "n_train = 975\n", "n_test = 525\n", "n_test_leavers = 71\n", "grown on 975 people, tested on 525 it has never seen (71 of whom left)\n" ] } ], "source": [ "#| caption: 65% to build the tree on, 35% it never sees\n", "from sklearn.model_selection import train_test_split\n", "\n", "Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35, random_state=7, stratify=y)\n", "record(\"n_train\", len(Xtr))\n", "record(\"n_test\", len(Xte))\n", "record(\"n_test_leavers\", int(yte.sum()))\n", "print(f\"grown on {len(Xtr)} people, tested on {len(Xte)} it has never seen \"\n", " f\"({int(yte.sum())} of whom left)\")" ] }, { "cell_type": "markdown", "id": "a6ac7ba8", "metadata": {}, "source": [ "## 4. Every depth from one to twenty\n", "\n", "**Depth** is how many questions the tree may ask before it has to decide. A\n", "depth of 3 means at most three questions about any one person.\n", "\n", "Build the tree at each depth from 1 to 20. Score each one twice: once on the\n", "people it learned from, once on the people it has never seen.\n", "\n", "Watch the two scores separate." ] }, { "cell_type": "code", "execution_count": 5, "id": "d63e97e6", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:30.792828Z", "iopub.status.busy": "2026-08-20T05:48:30.792678Z", "iopub.status.idle": "2026-08-20T05:48:31.116412Z", "shell.execute_reply": "2026-08-20T05:48:31.116152Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "depth leaves on what it saw on what it did not\n", " 1 2 0.866 0.865\n", " 2 4 0.866 0.865\n", " 3 8 0.868 0.867\n", " 4 15 0.884 0.855\n", " 5 26 0.896 0.850\n", " 6 45 0.916 0.838\n", " 7 67 0.942 0.834\n", " 8 85 0.962 0.827\n", " 9 100 0.971 0.808\n", " 10 109 0.982 0.796\n", " 11 115 0.987 0.796\n", " 12 121 0.992 0.794\n", " 13 125 0.998 0.783\n", " 14 127 0.999 0.785\n", " 15 129 1.000 0.792\n", " 16 129 1.000 0.792\n", " 17 129 1.000 0.792\n", " 18 129 1.000 0.792\n", " 19 129 1.000 0.792\n", " 20 129 1.000 0.792\n", "sweep = [{'depth': 1, 'train': 0.866, 'test': 0.865, 'leaves': 2, 'flagged': 0, 'caught': 0}, {'depth': 2, 'train': 0.866, 'test': 0.865, 'leaves': 4, 'flagged': 0, 'caught': 0}, {'depth': 3, 'train': 0.868, 'test': 0.867, 'leaves': 8, 'flagged': 1, 'caught': 1}, {'depth': 4, 'train': 0.884, 'test': 0.855, 'leaves': 15, 'flagged': 41, 'caught': 18}, {'depth': 5, 'train': 0.896, 'test': 0.85, 'leaves': 26, 'flagged': 16, 'caught': 4}, {'depth': 6, 'train': 0.916, 'test': 0.838, 'leaves': 45, 'flagged': 28, 'caught': 7}, {'depth': 7, 'train': 0.942, 'test': 0.834, 'leaves': 67, 'flagged': 46, 'caught': 15}, {'depth': 8, 'train': 0.962, 'test': 0.827, 'leaves': 85, 'flagged': 50, 'caught': 15}, {'depth': 9, 'train': 0.971, 'test': 0.808, 'leaves': 100, 'flagged': 48, 'caught': 9}, {'depth': 10, 'train': 0.982, 'test': 0.796, 'leaves': 109, 'flagged': 58, 'caught': 11}, {'depth': 11, 'train': 0.987, 'test': 0.796, 'leaves': 115, 'flagged': 68, 'caught': 16}, {'depth': 12, 'train': 0.992, 'test': 0.794, 'leaves': 121, 'flagged': 67, 'caught': 15}, {'depth': 13, 'train': 0.998, 'test': 0.783, 'leaves': 125, 'flagged': 69, 'caught': 13}, {'depth': 14, 'train': 0.999, 'test': 0.785, 'leaves': 127, 'flagged': 68, 'caught': 13}, {'depth': 15, 'train': 1.0, 'test': 0.792, 'leaves': 129, 'flagged': 68, 'caught': 15}, {'depth': 16, 'train': 1.0, 'test': 0.792, 'leaves': 129, 'flagged': 68, 'caught': 15}, {'depth': 17, 'train': 1.0, 'test': 0.792, 'leaves': 129, 'flagged': 68, 'caught': 15}, {'depth': 18, 'train': 1.0, 'test': 0.792, 'leaves': 129, 'flagged': 68, 'caught': 15}, {'depth': 19, 'train': 1.0, 'test': 0.792, 'leaves': 129, 'flagged': 68, 'caught': 15}, {'depth': 20, 'train': 1.0, 'test': 0.792, 'leaves': 129, 'flagged': 68, 'caught': 15}]\n", "deep_train = 1.0\n", "deep_test = 0.792\n", "deep_leaves = 129\n", "best_depth = 3\n", "best_test = 0.867\n", "gap_at_deepest = 0.208\n", "deep_train_pct = 100.0%\n", "deep_test_pct = 79.2%\n", "gap_pct = 20.8%\n", "best_test_pct = 86.7%\n", "figure learned-and-memorised -> perfect-on-what-it-has-seen.learned-and-memorised.{light,dark}.svg\n" ] } ], "source": [ "#| caption: Two scores at every depth: on the training people, and on the held-out people\n", "from sklearn.tree import DecisionTreeClassifier\n", "\n", "DEPTHS = list(range(1, 21))\n", "rows = []\n", "for d in DEPTHS:\n", " t = DecisionTreeClassifier(max_depth=d, criterion=\"gini\", random_state=7).fit(Xtr, ytr)\n", " pred = t.predict(Xte)\n", " rows.append({\n", " \"depth\": d,\n", " \"train\": round(float(t.score(Xtr, ytr)), 3),\n", " \"test\": round(float(t.score(Xte, yte)), 3),\n", " \"leaves\": int(t.get_n_leaves()),\n", " \"flagged\": int((pred == 1).sum()),\n", " \"caught\": int(((pred == 1) & (yte == 1)).sum()),\n", " })\n", "\n", "print(f\"{'depth':>5} {'leaves':>7} {'on what it saw':>16} {'on what it did not':>20}\")\n", "for r in rows:\n", " print(f\"{r['depth']:>5} {r['leaves']:>7} {r['train']:>16.3f} {r['test']:>20.3f}\")\n", "\n", "record(\"sweep\", rows)\n", "deepest = rows[-1]\n", "record(\"deep_train\", deepest[\"train\"])\n", "record(\"deep_test\", deepest[\"test\"])\n", "record(\"deep_leaves\", deepest[\"leaves\"])\n", "best = max(rows, key=lambda r: r[\"test\"])\n", "record(\"best_depth\", best[\"depth\"])\n", "record(\"best_test\", best[\"test\"])\n", "record(\"gap_at_deepest\", round(deepest[\"train\"] - deepest[\"test\"], 3))\n", "record(\"deep_train_pct\", f\"{deepest['train']:.1%}\")\n", "record(\"deep_test_pct\", f\"{deepest['test']:.1%}\")\n", "record(\"gap_pct\", f\"{deepest['train'] - deepest['test']:.1%}\")\n", "record(\"best_test_pct\", f\"{best['test']:.1%}\")\n", "\n", "def plot(ax):\n", " ax.plot([r[\"depth\"] for r in rows], [r[\"train\"] for r in rows],\n", " marker=\"o\", ms=4, color=\"#9aa0aa\", label=\"on the people it learned from\")\n", " ax.plot([r[\"depth\"] for r in rows], [r[\"test\"] for r in rows],\n", " marker=\"o\", ms=4, color=\"#e2574c\", label=\"on the people it has never met\")\n", " ax.axhline(baseline, ls=\"--\", lw=1.2, color=\"#3b6fd4\")\n", " ax.text(20, baseline + 0.006, 'saying \"stays\" to everybody', ha=\"right\", fontsize=9, color=\"#3b6fd4\")\n", " ax.set_xlabel(\"How many questions deep the tree may go\")\n", " ax.set_ylabel(\"Share called right\")\n", " ax.set_xticks([1, 5, 10, 15, 20])\n", " ax.set_ylim(0.75, 1.02)\n", " ax.legend(frameon=False, fontsize=9, loc=\"center right\")\n", "\n", "save_fig(\"learned-and-memorised\", plot, figsize=(7, 4.0))" ] }, { "cell_type": "markdown", "id": "13564c47", "metadata": {}, "source": [ "## 5. The gap is the memorising\n", "\n", "Subtract the second score from the first. The difference is the part of the\n", "score that only exists on the training rows.\n", "\n", "There is a name for this. When a model does much better on its training rows\n", "than on new data, it has **overfitted**. It has stored details of those\n", "particular rows rather than found a pattern that holds generally.\n", "\n", "Two things to notice in the numbers below. The gap grows with depth. And the\n", "deep trees end up scoring *below* the baseline — worse than saying \"stays\" to\n", "everyone." ] }, { "cell_type": "code", "execution_count": 6, "id": "8f4e6fb2", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:31.117630Z", "iopub.status.busy": "2026-08-20T05:48:31.117508Z", "iopub.status.idle": "2026-08-20T05:48:31.163936Z", "shell.execute_reply": "2026-08-20T05:48:31.163682Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "figure the-gap -> perfect-on-what-it-has-seen.the-gap.{light,dark}.svg\n", "first_depth_below_baseline = 4\n", "n_depths_beating_baseline = 1\n", "depths that beat 'stays for everybody': 1 of 20\n", "best of them: depth 3 at 0.867, against a free 0.865\n" ] } ], "source": [ "#| caption: How much of each score exists only on the training people\n", "def plot2(ax):\n", " gaps = [r[\"train\"] - r[\"test\"] for r in rows]\n", " ax.bar([r[\"depth\"] for r in rows], gaps, color=\"#e2574c\", width=0.62)\n", " ax.set_xlabel(\"How many questions deep the tree may go\")\n", " ax.set_ylabel(\"Score that exists only\\non the training set\")\n", " ax.set_xticks([1, 5, 10, 15, 20])\n", " ax.grid(axis=\"x\", visible=False)\n", "\n", "save_fig(\"the-gap\", plot2, figsize=(7, 3.2))\n", "\n", "below = [r for r in rows if r[\"test\"] < baseline]\n", "record(\"first_depth_below_baseline\", below[0][\"depth\"] if below else None)\n", "record(\"n_depths_beating_baseline\", sum(1 for r in rows if r[\"test\"] > baseline))\n", "print(f\"depths that beat 'stays for everybody': {sum(1 for r in rows if r['test'] > baseline)} of {len(rows)}\")\n", "print(f\"best of them: depth {best['depth']} at {best['test']:.3f}, against a free {baseline:.3f}\")" ] }, { "cell_type": "markdown", "id": "3959f3e1", "metadata": {}, "source": [ "## 6. The shallow tree, drawn\n", "\n", "So use a shallow tree instead. Two questions deep, four leaves, and a score\n", "just above the baseline.\n", "\n", "Now read what it actually says. The score does not tell you." ] }, { "cell_type": "code", "execution_count": 7, "id": "6ff3fdce", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:31.165472Z", "iopub.status.busy": "2026-08-20T05:48:31.165386Z", "iopub.status.idle": "2026-08-20T05:48:31.181573Z", "shell.execute_reply": "2026-08-20T05:48:31.181236Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shallow_tree = {'q': 'Works overtime?', 'note': '975 people - 131 left', 'yes': {'q': 'Job satisfaction under 3?', 'note': '308 people - 75 left', 'yes': {'leaf': 'stays', 'note': '107 people - 43 left'}, 'no': {'leaf': 'stays', 'note': '201 people - 32 left'}}, 'no': {'q': 'Stock option level under 1?', 'note': '667 people - 56 left', 'yes': {'leaf': 'stays', 'note': '223 people - 30 left'}, 'no': {'leaf': 'stays', 'note': '444 people - 26 left'}}}\n", "shallow_test = 0.865\n", "shallow_test_pct = 86.5%\n", "shallow_flagged = 0\n", "shallow_tree = {'q': 'Works overtime?', 'note': '975 people - 131 left', 'yes': {'q': 'Job satisfaction under 3?', 'note': '308 people - 75 left', 'yes': {'leaf': 'stays', 'note': '107 people - 43 left'}, 'no': {'leaf': 'stays', 'note': '201 people - 32 left'}}, 'no': {'q': 'Stock option level under 1?', 'note': '667 people - 56 left', 'yes': {'leaf': 'stays', 'note': '223 people - 30 left'}, 'no': {'leaf': 'stays', 'note': '444 people - 26 left'}}}\n", "worst_leaf_people = 107\n", "worst_leaf_left = 43\n", "worst_leaf_share = 0.4\n", "worst_leaf_pct = 40%\n", "worst_leaf_call = stays\n", "depth 2 scores 0.865 against a free 0.865\n", "and flags 0 people out of 525 for a conversation\n" ] } ], "source": [ "#| caption: The two-question tree, turned into something readable\n", "import numpy as np\n", "\n", "LABEL = {\n", " \"MonthlyIncome\": \"Monthly income\", \"Age\": \"Age\", \"DistanceFromHome\": \"Distance from home\",\n", " \"YearsAtCompany\": \"Years at company\", \"TotalWorkingYears\": \"Total working years\",\n", " \"YearsSinceLastPromotion\": \"Years since promotion\", \"YearsInCurrentRole\": \"Years in role\",\n", " \"YearsWithCurrManager\": \"Years with manager\", \"StockOptionLevel\": \"Stock option level\",\n", " \"JobLevel\": \"Job level\", \"EnvironmentSatisfaction\": \"Environment score\",\n", " \"JobSatisfaction\": \"Job satisfaction\", \"WorkLifeBalance\": \"Work-life balance\",\n", " \"TrainingTimesLastYear\": \"Trainings last year\", \"NumCompaniesWorked\": \"Employers before this\",\n", " \"PercentSalaryHike\": \"Last salary hike %\", \"RelationshipSatisfaction\": \"Relationship score\",\n", " \"Education\": \"Education level\", \"PerformanceRating\": \"Performance rating\",\n", " \"OverTime_Yes\": \"Works overtime?\", \"Gender_Male\": \"Male?\",\n", " \"Department_R&D\": \"In R and D?\", \"Department_Sales\": \"In Sales?\",\n", "}\n", "\n", "def to_nodes(clf, names, node=0):\n", " \"\"\"sklearn's arrays -> the {q, yes, no, note} shape the lesson draws.\n", "\n", " sklearn always sends `feature <= threshold` LEFT. For a yes/no column that\n", " means left is the NO side, so the branches are swapped and the question is\n", " asked the way a person would ask it. Getting this backwards draws a tree\n", " that is a mirror image of the one that was fitted, and nothing about the\n", " picture would look wrong.\n", " \"\"\"\n", " t = clf.tree_\n", " stayed, left_co = t.value[node][0]\n", " n = int(t.n_node_samples[node])\n", " share = left_co / (stayed + left_co)\n", " note = f\"{n} people - {round(share * n)} left\"\n", " if t.children_left[node] == -1:\n", " return {\"leaf\": \"leaves\" if share >= 0.5 else \"stays\", \"note\": note}\n", " col = names[t.feature[node]]\n", " thr = t.threshold[node]\n", " lo = to_nodes(clf, names, t.children_left[node])\n", " hi = to_nodes(clf, names, t.children_right[node])\n", " if col.endswith(\"_Yes\") or col.startswith((\"Gender_\", \"Department_\")):\n", " return {\"q\": LABEL.get(col, col), \"note\": note, \"yes\": hi, \"no\": lo}\n", " # A threshold of 0.5 on a whole-number column reads as \"under 0\" if it is\n", " # simply rounded, which is not what the tree does. Round UP: \"under 1\"\n", " # means zero, which is the split that was actually made.\n", " whole = bool(X[col].dropna().mod(1).eq(0).all())\n", " step = np.ceil(thr) if whole else thr\n", " return {\"q\": f\"{LABEL.get(col, col)} under {step:,.0f}?\", \"note\": note, \"yes\": lo, \"no\": hi}\n", "\n", "shallow = DecisionTreeClassifier(max_depth=2, criterion=\"gini\", random_state=7).fit(Xtr, ytr)\n", "record(\"shallow_tree\", to_nodes(shallow, list(X.columns)))\n", "record(\"shallow_test\", round(float(shallow.score(Xte, yte)), 3))\n", "record(\"shallow_test_pct\", f\"{shallow.score(Xte, yte):.1%}\")\n", "record(\"shallow_flagged\", int((shallow.predict(Xte) == 1).sum()))\n", "\n", "# The leaf a manager should be most alarmed by, and the tree calls it \"stays\".\n", "def worst_leaf(node):\n", " if \"leaf\" in node:\n", " n, left = int(node[\"note\"].split()[0]), int(node[\"note\"].split(\"- \")[1].split()[0])\n", " return (left / n, n, left, node[\"leaf\"])\n", " return max(worst_leaf(node[\"yes\"]), worst_leaf(node[\"no\"]))\n", "\n", "share, n_leaf, n_left_leaf, call = worst_leaf(record(\"shallow_tree\", to_nodes(shallow, list(X.columns))))\n", "record(\"worst_leaf_people\", n_leaf)\n", "record(\"worst_leaf_left\", n_left_leaf)\n", "record(\"worst_leaf_share\", round(share, 2))\n", "record(\"worst_leaf_pct\", f\"{share:.0%}\")\n", "record(\"worst_leaf_call\", call)\n", "print(f\"depth 2 scores {shallow.score(Xte, yte):.3f} against a free {baseline:.3f}\")\n", "print(f\"and flags {int((shallow.predict(Xte) == 1).sum())} people out of {len(yte)} for a conversation\")" ] }, { "cell_type": "markdown", "id": "0fe2df7a", "metadata": {}, "source": [ "## 7. What the score was hiding\n", "\n", "The shallow tree beats the baseline. It also flags nobody: all four of its\n", "leaves say \"stays\", because in every leaf the stayers outnumber the leavers.\n", "\n", "So it is 86.7% accurate and it never names a single person to talk to.\n", "\n", "The tree at depth 4 scores *below* the baseline and finds 18 real leavers. Count\n", "the two trees by leavers found and the order reverses. Accuracy is not measuring\n", "what we want." ] }, { "cell_type": "code", "execution_count": 8, "id": "a0da3c22", "metadata": { "execution": { "iopub.execute_input": "2026-08-20T05:48:31.182904Z", "iopub.status.busy": "2026-08-20T05:48:31.182822Z", "iopub.status.idle": "2026-08-20T05:48:31.234529Z", "shell.execute_reply": "2026-08-20T05:48:31.234303Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "depth test score people flagged leavers caught\n", " 1 0.865 0 0 of 71\n", " 2 0.865 0 0 of 71\n", " 3 0.867 1 1 of 71\n", " 4 0.855 41 18 of 71\n", " 5 0.850 16 4 of 71\n", " 6 0.838 28 7 of 71\n", " 7 0.834 46 15 of 71\n", " 8 0.827 50 15 of 71\n", "useful_depth = 4\n", "useful_test = 0.855\n", "useful_test_pct = 85.5%\n", "useful_flagged = 41\n", "useful_caught = 18\n", "figure what-it-catches -> perfect-on-what-it-has-seen.what-it-catches.{light,dark}.svg\n" ] } ], "source": [ "#| caption: How many real leavers each tree finds\n", "print(f\"{'depth':>5} {'test score':>11} {'people flagged':>15} {'leavers caught':>15}\")\n", "for r in rows[:8]:\n", " print(f\"{r['depth']:>5} {r['test']:>11.3f} {r['flagged']:>15} {r['caught']:>10} of {int(yte.sum())}\")\n", "\n", "useful = max(rows, key=lambda r: r[\"caught\"])\n", "record(\"useful_depth\", useful[\"depth\"])\n", "record(\"useful_test\", useful[\"test\"])\n", "record(\"useful_test_pct\", f\"{useful['test']:.1%}\")\n", "record(\"useful_flagged\", useful[\"flagged\"])\n", "record(\"useful_caught\", useful[\"caught\"])\n", "\n", "def plot3(ax):\n", " ax.bar([r[\"depth\"] for r in rows], [r[\"caught\"] for r in rows], color=\"#e2574c\", width=0.62)\n", " ax.axhline(int(yte.sum()), ls=\"--\", lw=1.2, color=\"#3b6fd4\")\n", " ax.text(20, int(yte.sum()) + 1.2, f\"all {int(yte.sum())} leavers in the test set\",\n", " ha=\"right\", fontsize=9, color=\"#3b6fd4\")\n", " ax.set_xlabel(\"How many questions deep the tree may go\")\n", " ax.set_ylabel(\"Leavers it actually found\")\n", " ax.set_xticks([1, 5, 10, 15, 20])\n", " ax.set_ylim(0, int(yte.sum()) * 1.18)\n", " ax.grid(axis=\"x\", visible=False)\n", "\n", "save_fig(\"what-it-catches\", plot3, figsize=(7, 3.2))" ] }, { "cell_type": "markdown", "id": "75ec42de", "metadata": {}, "source": [ "## Try this\n", "\n", "1. **Change the seed** on the split. Set `random_state=8`, then 9, then 10, and\n", " run it again. The two lines keep the same shape. The best depth moves around.\n", " That is why you should not pick a depth from one split.\n", "2. **Limit the leaf size instead of the depth.** Use `min_samples_leaf=20`.\n", " Compare the gap at depth 20 with and without it.\n", "3. **Use only 200 people to build the tree.** Both scores get worse. Watch which\n", " of the two moves further. That tells you what more data would and would not\n", " fix." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 5 }