{ "cells": [ { "cell_type": "markdown", "id": "348fb7d6", "metadata": {}, "source": [ "# Decision Trees · Turning a tree into a list of names\n", "\n", "Every lesson so far has ended in a number. This one ends in a list of people\n", "and a sentence to open with, which is the only form any of this can actually be\n", "used in.\n", "\n", "The tree from lesson 6 is the one being used — no employee number, nothing\n", "invented. It scores **below** the free number and it is the first model in this\n", "path worth having, and by the end of this notebook that will stop sounding like\n", "a contradiction." ] }, { "cell_type": "code", "execution_count": 1, "id": "9205874b", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T14:00:43.672607Z", "iopub.status.busy": "2026-08-19T14:00:43.672395Z", "iopub.status.idle": "2026-08-19T14:00:43.959598Z", "shell.execute_reply": "2026-08-19T14:00:43.959316Z" } }, "outputs": [], "source": [ "# hide — plumbing so this file runs both in the repo and in Colab.\n", "try:\n", " from _figkit import save_fig, record\n", "except ImportError: # Colab — no repo, no problem\n", " def save_fig(name, plot, **kw):\n", " import matplotlib.pyplot as plt\n", " fig, ax = plt.subplots(figsize=kw.get('figsize', (7, 4.2))); plot(ax); plt.show()\n", " def record(key, value):\n", " print(f'{key} = {value}'); return value" ] }, { "cell_type": "markdown", "id": "ada993d6", "metadata": {}, "source": [ "## 1. The tree, as it was left\n", "\n", "Same file, same split, same depth. Nothing new is fitted in this lesson: the\n", "whole point is what you do with a model you already have." ] }, { "cell_type": "code", "execution_count": 2, "id": "f025ac1c", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T14:00:43.961149Z", "iopub.status.busy": "2026-08-19T14:00:43.961012Z", "iopub.status.idle": "2026-08-19T14:00:44.894663Z", "shell.execute_reply": "2026-08-19T14:00:44.894392Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "n_test = 525\n", "n_leavers = 71\n", "base_rate = 13.5%\n", "525 people the tree has never seen, 71 of whom left (13.5%)\n" ] } ], "source": [ "#| caption: The honest tree from lesson 6 — no employee number, nothing invented\n", "from pathlib import Path\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.tree import DecisionTreeClassifier\n", "\n", "REL = 'public/datasets/decision-trees/hr-employee-attrition.csv'\n", "here = Path.cwd()\n", "LOCAL = next((p / REL for p in [here, *here.parents] if (p / REL).exists()), None)\n", "CSV = LOCAL or 'https://raw.githubusercontent.com/Dr-Shashank-S-Sharma/expedify-ai-courses/main/datasets/decision-trees/hr-employee-attrition.csv'\n", "\n", "df = pd.read_csv(CSV)\n", "y = (df.Attrition == \"Yes\").astype(int)\n", "X = pd.get_dummies(df.drop(columns=[\"Attrition\", \"EmployeeID\"]), drop_first=True)\n", "\n", "tr, te = train_test_split(range(len(df)), test_size=0.35, random_state=7, stratify=y)\n", "ytr, yte = y.iloc[tr], y.iloc[te].values\n", "\n", "tree = DecisionTreeClassifier(max_depth=4, random_state=7).fit(X.iloc[tr], ytr)\n", "risk = tree.predict_proba(X.iloc[te])[:, 1]\n", "leaf = tree.apply(X.iloc[te])\n", "\n", "record(\"n_test\", len(te))\n", "record(\"n_leavers\", int(yte.sum()))\n", "record(\"base_rate\", f\"{yte.mean():.1%}\")\n", "print(f\"{len(te)} people the tree has never seen, {int(yte.sum())} of whom left \"\n", " f\"({yte.mean():.1%})\")" ] }, { "cell_type": "markdown", "id": "60d6b8ae", "metadata": {}, "source": [ "## 2. A leaf is a probability, not a verdict\n", "\n", "`predict` gives you \"stays\" or \"leaves\". That is not what the tree knows.\n", "\n", "Every leaf holds a mix, and the share of leavers in it is a **risk** — 1% in one\n", "leaf, 59% in another. Calling everything under half \"stays\" throws all of that\n", "away, and it is the whole reason lesson 6's shallow tree flagged nobody." ] }, { "cell_type": "code", "execution_count": 3, "id": "9b3fe2b8", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T14:00:44.895937Z", "iopub.status.busy": "2026-08-19T14:00:44.895798Z", "iopub.status.idle": "2026-08-19T14:00:44.900902Z", "shell.execute_reply": "2026-08-19T14:00:44.900659Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " the leaf says people in it who actually left\n", " 100% 4 0\n", " 100% 1 0\n", " 100% 2 1\n", " 100% 1 1\n", " 59% 33 16\n", " 46% 7 2\n", " 27% 14 6\n", " 26% 9 0\n", " 22% 33 4\n", " 13% 99 14\n", " 11% 59 7\n", " 8% 11 4\n", " 7% 63 5\n", " 6% 84 8\n", " 1% 105 3\n", "leaf_rows = [{'leaf': 8, 'risk': 1.0, 'people': 4, 'left': 0}, {'leaf': 12, 'risk': 1.0, 'people': 1, 'left': 0}, {'leaf': 23, 'risk': 1.0, 'people': 2, 'left': 1}, {'leaf': 25, 'risk': 1.0, 'people': 1, 'left': 1}, {'leaf': 19, 'risk': 0.59, 'people': 33, 'left': 16}, {'leaf': 28, 'risk': 0.46, 'people': 7, 'left': 2}, {'leaf': 20, 'risk': 0.27, 'people': 14, 'left': 6}, {'leaf': 4, 'risk': 0.26, 'people': 9, 'left': 0}, {'leaf': 7, 'risk': 0.22, 'people': 33, 'left': 4}, {'leaf': 27, 'risk': 0.13, 'people': 99, 'left': 14}, {'leaf': 11, 'risk': 0.11, 'people': 59, 'left': 7}, {'leaf': 22, 'risk': 0.08, 'people': 11, 'left': 4}, {'leaf': 14, 'risk': 0.07, 'people': 63, 'left': 5}, {'leaf': 5, 'risk': 0.06, 'people': 84, 'left': 8}, {'leaf': 15, 'risk': 0.01, 'people': 105, 'left': 3}]\n", "big_leaf_risk = 59%\n", "big_leaf_people = 33\n", "big_leaf_left = 16\n", "n_leaves_shown = 15\n" ] }, { "data": { "text/plain": [ "15" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| caption: What each leaf of the tree actually says about the people who land in it\n", "rows = []\n", "for lf in sorted(set(leaf), key=lambda l: -risk[leaf == l][0]):\n", " m = leaf == lf\n", " rows.append({\"leaf\": int(lf), \"risk\": round(float(risk[m][0]), 2),\n", " \"people\": int(m.sum()), \"left\": int(yte[m].sum())})\n", "\n", "print(f\"{'the leaf says':>14} {'people in it':>13} {'who actually left':>19}\")\n", "for r in rows:\n", " print(f\"{r['risk']:>13.0%} {r['people']:>13} {r['left']:>19}\")\n", "\n", "record(\"leaf_rows\", rows)\n", "big = max((r for r in rows if r[\"people\"] >= 20), key=lambda r: r[\"risk\"])\n", "record(\"big_leaf_risk\", f\"{big['risk']:.0%}\")\n", "record(\"big_leaf_people\", big[\"people\"])\n", "record(\"big_leaf_left\", big[\"left\"])\n", "record(\"n_leaves_shown\", len(rows))" ] }, { "cell_type": "markdown", "id": "8e6b430e", "metadata": {}, "source": [ "## 3. Rank, do not classify\n", "\n", "Once a leaf is a risk, the useful move is obvious and it is not the one the\n", "library does by default: **sort everybody by it and start at the top.**\n", "\n", "You are no longer asking \"who will leave\". You are asking \"if I can have thirty\n", "conversations this quarter, whose?\"" ] }, { "cell_type": "code", "execution_count": 4, "id": "4513efc0", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T14:00:44.902205Z", "iopub.status.busy": "2026-08-19T14:00:44.902083Z", "iopub.status.idle": "2026-08-19T14:00:44.984900Z", "shell.execute_reply": "2026-08-19T14:00:44.984466Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " conversations leavers reached if chosen at random lift\n", " 10 4 1.4 3.0x\n", " 25 10 3.4 3.0x\n", " 50 21 6.8 3.1x\n", " 75 26 10.1 2.6x\n", " 100 30 13.5 2.2x\n", " 150 36 20.3 1.8x\n", " 200 44 27.0 1.6x\n", "budget_rows = [{'budget': 10, 'found': 4, 'random': 1.4, 'lift': np.float64(3.0)}, {'budget': 25, 'found': 10, 'random': 3.4, 'lift': np.float64(3.0)}, {'budget': 50, 'found': 21, 'random': 6.8, 'lift': np.float64(3.1)}, {'budget': 75, 'found': 26, 'random': 10.1, 'lift': np.float64(2.6)}, {'budget': 100, 'found': 30, 'random': 13.5, 'lift': np.float64(2.2)}, {'budget': 150, 'found': 36, 'random': 20.3, 'lift': np.float64(1.8)}, {'budget': 200, 'found': 44, 'random': 27.0, 'lift': np.float64(1.6)}]\n", "fifty_found = 21\n", "fifty_random = 6.8\n", "fifty_lift = 3.1x\n", "share_of_leavers_at_fifty = 30%\n", "figure working-down-the-list -> a-list-of-names.working-down-the-list.{light,dark}.svg\n" ] } ], "source": [ "#| caption: Work down the ranked list and count how many leavers you reach\n", "order = np.argsort(-risk, kind=\"stable\")\n", "found = np.cumsum(yte[order])\n", "\n", "budgets = [10, 25, 50, 75, 100, 150, 200]\n", "print(f\"{'conversations':>14} {'leavers reached':>17} {'if chosen at random':>21} {'lift':>7}\")\n", "table = []\n", "for k in budgets:\n", " at_random = k * float(yte.mean())\n", " table.append({\"budget\": k, \"found\": int(found[k - 1]),\n", " \"random\": round(at_random, 1), \"lift\": round(found[k - 1] / at_random, 1)})\n", " print(f\"{k:>14} {int(found[k-1]):>17} {at_random:>21.1f} {found[k-1]/at_random:>6.1f}x\")\n", "\n", "record(\"budget_rows\", table)\n", "FIFTY = next(r for r in table if r[\"budget\"] == 50)\n", "record(\"fifty_found\", FIFTY[\"found\"])\n", "record(\"fifty_random\", FIFTY[\"random\"])\n", "record(\"fifty_lift\", f\"{FIFTY['lift']}x\")\n", "record(\"share_of_leavers_at_fifty\", f\"{FIFTY['found'] / yte.sum():.0%}\")\n", "\n", "def plot(ax):\n", " n = len(order)\n", " ax.plot(range(1, n + 1), found, color=\"#e2574c\", label=\"working down the tree's ranking\")\n", " ax.plot([0, n], [0, yte.sum()], ls=\"--\", lw=1.2, color=\"#9aa0aa\", label=\"picking people at random\")\n", " ax.plot([0, yte.sum(), n], [0, yte.sum(), yte.sum()], ls=\":\", lw=1.2, color=\"#3b6fd4\",\n", " label=\"a perfect list\")\n", " ax.axvline(50, lw=1, color=\"#c9c9d1\")\n", " ax.annotate(f\"50 conversations\\n{int(found[49])} leavers reached\", (50, found[49]),\n", " textcoords=\"offset points\", xytext=(14, -6), fontsize=9, color=\"#77777f\")\n", " ax.set_xlabel(\"Conversations you are willing to have\")\n", " ax.set_ylabel(\"Leavers you reach\")\n", " ax.legend(frameon=False, fontsize=9, loc=\"lower right\")\n", "\n", "save_fig(\"working-down-the-list\", plot, figsize=(7, 4.0))" ] }, { "cell_type": "markdown", "id": "0e4353c8", "metadata": {}, "source": [ "## 4. Where the top of the list comes from\n", "\n", "Look again at the leaf table. Four of the leaves say **100%** — and between them\n", "they hold a handful of people.\n", "\n", "Those are lesson 6's overfitting, arriving in the action list. A leaf built on\n", "three training rows will happily claim certainty, and ranking puts it first.\n", "The leaf that actually earns its place is the big one." ] }, { "cell_type": "code", "execution_count": 5, "id": "bfc0504f", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T14:00:44.986384Z", "iopub.status.busy": "2026-08-19T14:00:44.986286Z", "iopub.status.idle": "2026-08-19T14:00:44.991031Z", "shell.execute_reply": "2026-08-19T14:00:44.990718Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "leaves claiming certainty: 4, holding 8 people between them, of whom 2 left\n", "the big leaf: 33 people, 16 left (48%)\n", "n_certain_leaves = 4\n", "certain_people = 8\n", "certain_left = 2\n" ] }, { "data": { "text/plain": [ "2" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| caption: The certain little leaves, against the one worth acting on\n", "tiny = [r for r in rows if r[\"risk\"] >= 0.99]\n", "print(f\"leaves claiming certainty: {len(tiny)}, holding {sum(r['people'] for r in tiny)} people \"\n", " f\"between them, of whom {sum(r['left'] for r in tiny)} left\")\n", "print(f\"the big leaf: {big['people']} people, {big['left']} left \"\n", " f\"({big['left'] / big['people']:.0%})\")\n", "\n", "record(\"n_certain_leaves\", len(tiny))\n", "record(\"certain_people\", sum(r[\"people\"] for r in tiny))\n", "record(\"certain_left\", sum(r[\"left\"] for r in tiny))" ] }, { "cell_type": "markdown", "id": "73a20728", "metadata": {}, "source": [ "## 5. What you would actually say\n", "\n", "This is where a tree earns the thing no score can give you: **the reason.**\n", "\n", "The big leaf is not an anonymous high-risk bucket. It is the end of a path of\n", "questions, and reading that path back is a sentence a manager can walk into a\n", "room with." ] }, { "cell_type": "code", "execution_count": 6, "id": "97311834", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T14:00:44.992191Z", "iopub.status.busy": "2026-08-19T14:00:44.992094Z", "iopub.status.idle": "2026-08-19T14:00:45.001094Z", "shell.execute_reply": "2026-08-19T14:00:45.000752Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "big_leaf_path = ['works overtime', 'job satisfaction under 3', 'age under 51', 'monthly income under 13,174']\n", " works overtime\n", " job satisfaction under 3\n", " age under 51\n", " monthly income under 13,174\n", "-> 33 such people in the held-out set, 16 of them left\n" ] } ], "source": [ "#| caption: The path to the biggest high-risk leaf, read back as a sentence\n", "LABEL = {\"OverTime_Yes\": \"works overtime\", \"JobSatisfaction\": \"job satisfaction\",\n", " \"MonthlyIncome\": \"monthly income\", \"Age\": \"age\", \"YearsAtCompany\": \"years at company\",\n", " \"TotalWorkingYears\": \"total working years\", \"StockOptionLevel\": \"stock option level\",\n", " \"DistanceFromHome\": \"distance from home\", \"EnvironmentSatisfaction\": \"environment score\",\n", " \"NumCompaniesWorked\": \"employers before this\", \"Education\": \"education level\",\n", " \"YearsSinceLastPromotion\": \"years since promotion\", \"JobLevel\": \"job level\",\n", " \"WorkLifeBalance\": \"work-life balance\", \"YearsWithCurrManager\": \"years with manager\",\n", " \"YearsInCurrentRole\": \"years in role\", \"TrainingTimesLastYear\": \"trainings last year\",\n", " \"PercentSalaryHike\": \"last salary hike %\", \"RelationshipSatisfaction\": \"relationship score\",\n", " \"PerformanceRating\": \"performance rating\", \"Gender_Male\": \"is male\",\n", " \"Department_R&D\": \"is in R and D\", \"Department_Sales\": \"is in Sales\"}\n", "\n", "# The leaf id travels in `rows`, so this is a lookup rather than a float\n", "# comparison against a rounded number — which is how the first version of this\n", "# cell died.\n", "target = big[\"leaf\"]\n", "\n", "def path_to(clf, names, want, node=0, acc=None):\n", " acc = acc or []\n", " t = clf.tree_\n", " if t.children_left[node] == -1:\n", " return acc if node == want else None\n", " col, thr = names[t.feature[node]], t.threshold[node]\n", " binary = set(np.unique(X[col])) <= {0, 1, True, False}\n", " whole = bool(pd.Series(X[col]).dropna().mod(1).eq(0).all())\n", " step = int(np.ceil(thr)) if whole else round(float(thr), 1)\n", " lo = f\"does not {LABEL.get(col, col)}\" if binary else f\"{LABEL.get(col, col)} under {step:,}\"\n", " hi = f\"{LABEL.get(col, col)}\" if binary else f\"{LABEL.get(col, col)} {step:,} or more\"\n", " return (path_to(clf, names, want, t.children_left[node], acc + [lo])\n", " or path_to(clf, names, want, t.children_right[node], acc + [hi]))\n", "\n", "steps = path_to(tree, list(X.columns), target)\n", "record(\"big_leaf_path\", steps)\n", "for s in steps:\n", " print(\" \", s)\n", "print(f\"-> {big['people']} such people in the held-out set, {big['left']} of them left\")" ] }, { "cell_type": "markdown", "id": "c8e1b93c", "metadata": {}, "source": [ "## 6. The honest limit\n", "\n", "One thing before you take that list anywhere.\n", "\n", "Refit the tree on 95% of the training rows, chosen at random, and do it thirty\n", "times. The **first question never changes** — it is overtime every single time.\n", "The **list of fifty names does**.\n", "\n", "That is not a flaw you can tune away. It is what a single tree is, and the fix\n", "is a different model — which is a later path, not this one." ] }, { "cell_type": "code", "execution_count": 7, "id": "71c2dab6", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T14:00:45.003003Z", "iopub.status.busy": "2026-08-19T14:00:45.002875Z", "iopub.status.idle": "2026-08-19T14:00:45.135345Z", "shell.execute_reply": "2026-08-19T14:00:45.135026Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "first question, over 30 refits: {'OverTime_Yes': 30}\n", "top-fifty list overlap: mean 78%, worst 64%\n", "first_question_stability = 30 of 30\n", "first_question = works overtime\n", "list_overlap_mean = 78%\n", "list_overlap_worst = 64%\n", "list_churn = 22%\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "figure the-list-moves -> a-list-of-names.the-list-moves.{light,dark}.svg\n" ] } ], "source": [ "#| caption: Drop 5% of the training rows, thirty times, and see what survives\n", "import collections\n", "rng = np.random.default_rng(0)\n", "firsts = collections.Counter()\n", "overlap = []\n", "top50 = set(order[:50])\n", "\n", "for _ in range(30):\n", " keep = rng.choice(len(tr), int(len(tr) * 0.95), replace=False)\n", " idx = [tr[j] for j in keep]\n", " t2 = DecisionTreeClassifier(max_depth=4, random_state=7).fit(X.iloc[idx], y.iloc[idx])\n", " firsts[X.columns[t2.tree_.feature[0]]] += 1\n", " r2 = t2.predict_proba(X.iloc[te])[:, 1]\n", " overlap.append(len(top50 & set(np.argsort(-r2, kind=\"stable\")[:50])) / 50)\n", "\n", "print(\"first question, over 30 refits:\", dict(firsts))\n", "print(f\"top-fifty list overlap: mean {np.mean(overlap):.0%}, worst {np.min(overlap):.0%}\")\n", "\n", "record(\"first_question_stability\", f\"{max(firsts.values())} of 30\")\n", "record(\"first_question\", LABEL.get(max(firsts, key=firsts.get), max(firsts, key=firsts.get)))\n", "record(\"list_overlap_mean\", f\"{np.mean(overlap):.0%}\")\n", "record(\"list_overlap_worst\", f\"{np.min(overlap):.0%}\")\n", "record(\"list_churn\", f\"{1 - np.mean(overlap):.0%}\")\n", "\n", "def plot2(ax):\n", " ax.hist(overlap, bins=8, color=\"#e2574c\")\n", " ax.axvline(float(np.mean(overlap)), ls=\"--\", lw=1.2, color=\"#3b6fd4\")\n", " ax.text(float(np.mean(overlap)), 7.4, f\" mean {np.mean(overlap):.0%}\", fontsize=9, color=\"#3b6fd4\")\n", " ax.set_xlabel(\"Share of the original fifty names still on the list\")\n", " ax.set_ylabel(\"Refits\")\n", " ax.grid(axis=\"x\", visible=False)\n", "\n", "save_fig(\"the-list-moves\", plot2, figsize=(7, 3.2))" ] }, { "cell_type": "markdown", "id": "32bba8e7", "metadata": {}, "source": [ "## Try this\n", "\n", "1. **Price it.** Say a retention conversation costs an hour and keeping somebody\n", " is worth twenty. Work out the budget where the ranked list stops paying — and\n", " notice that nothing about the model changed, only the two numbers you chose.\n", "2. **Rank with the deeper tree.** Depth 10 scores far worse on accuracy. Does its\n", " ranked list find more leavers in the top fifty, or fewer?\n", "3. **Take the tiny certain leaves out** of the ranking entirely — drop any leaf\n", " built on fewer than ten training rows. The top of the list changes. Does the\n", " number of leavers you reach at fifty get better or worse?" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 5 }