{ "cells": [ { "cell_type": "markdown", "id": "7a6be5df", "metadata": {}, "source": [ "# Decision Trees · The column that gives the game away\n", "\n", "Last lesson dropped one column on the way in — the employee ID — and said the\n", "reason would be worth waiting for.\n", "\n", "Put it back. Nothing else changes: same people, same split, same tree.\n", "\n", "Two things come out of this, and the second is the more dangerous by a long\n", "way. **The tree uses the ID.** And there is a kind of broken model that a\n", "held-out set cannot catch at all." ] }, { "cell_type": "code", "execution_count": 1, "id": "687af166", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:53.556667Z", "iopub.status.busy": "2026-08-19T13:19:53.556542Z", "iopub.status.idle": "2026-08-19T13:19:53.820052Z", "shell.execute_reply": "2026-08-19T13:19:53.819592Z" } }, "outputs": [], "source": [ "# hide — plumbing so this file runs both in the repo and in Colab.\n", "try:\n", " from _figkit import save_fig, record\n", "except ImportError: # Colab — no repo, no problem\n", " def save_fig(name, plot, **kw):\n", " import matplotlib.pyplot as plt\n", " fig, ax = plt.subplots(figsize=kw.get('figsize', (7, 4.2))); plot(ax); plt.show()\n", " def record(key, value):\n", " print(f'{key} = {value}'); return value" ] }, { "cell_type": "markdown", "id": "15c14f40", "metadata": {}, "source": [ "## 1. The same setup as lesson 6\n", "\n", "Same file, same stratified split, same seed — so every number here is\n", "comparable with the last lesson's rather than merely similar to it." ] }, { "cell_type": "code", "execution_count": 2, "id": "315f38b5", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:53.825792Z", "iopub.status.busy": "2026-08-19T13:19:53.825347Z", "iopub.status.idle": "2026-08-19T13:19:54.943816Z", "shell.execute_reply": "2026-08-19T13:19:54.943577Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "975 people to learn from, 525 held back, 71 of whom left\n" ] } ], "source": [ "#| caption: The same people and the same split as the last lesson — with the ID left in this time\n", "from pathlib import Path\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.tree import DecisionTreeClassifier\n", "\n", "REL = 'public/datasets/decision-trees/hr-employee-attrition.csv'\n", "here = Path.cwd()\n", "LOCAL = next((p / REL for p in [here, *here.parents] if (p / REL).exists()), None)\n", "CSV = LOCAL or 'https://raw.githubusercontent.com/Dr-Shashank-S-Sharma/expedify-ai-courses/main/datasets/decision-trees/hr-employee-attrition.csv'\n", "\n", "df = pd.read_csv(CSV)\n", "y = (df.Attrition == \"Yes\").astype(int)\n", "WITHOUT_ID = pd.get_dummies(df.drop(columns=[\"Attrition\", \"EmployeeID\"]), drop_first=True)\n", "WITH_ID = pd.get_dummies(df.drop(columns=[\"Attrition\"]), drop_first=True)\n", "\n", "tr, te = train_test_split(range(len(df)), test_size=0.35, random_state=7, stratify=y)\n", "ytr, yte = y.iloc[tr], y.iloc[te]\n", "\n", "def fit(X, depth=4):\n", " return DecisionTreeClassifier(max_depth=depth, random_state=7).fit(X.iloc[tr], ytr)\n", "\n", "def report(X, depth=4):\n", " t = fit(X, depth)\n", " pred = t.predict(X.iloc[te])\n", " return {\n", " \"train\": round(float(t.score(X.iloc[tr], ytr)), 3),\n", " \"test\": round(float(t.score(X.iloc[te], yte)), 3),\n", " \"caught\": int(((pred == 1) & (yte == 1)).sum()),\n", " \"importances\": dict(sorted(zip(X.columns, t.feature_importances_), key=lambda r: -r[1])),\n", " }\n", "\n", "print(f\"{len(tr)} people to learn from, {len(te)} held back, {int(yte.sum())} of whom left\")" ] }, { "cell_type": "markdown", "id": "15416034", "metadata": {}, "source": [ "## 2. The tree uses the employee number\n", "\n", "Grow it at depth three, with the ID available, and ask how much of its decision\n", "making rests on that column.\n", "\n", "It is not zero. Nobody made a mistake, nobody wrote a bad line of code, and a\n", "few per cent of this model's reasoning is an employee number." ] }, { "cell_type": "code", "execution_count": 3, "id": "e0e1eaa9", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:54.944973Z", "iopub.status.busy": "2026-08-19T13:19:54.944886Z", "iopub.status.idle": "2026-08-19T13:19:54.976169Z", "shell.execute_reply": "2026-08-19T13:19:54.975957Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "depth 3 share of the tree's decisions resting on EmployeeID: 5.3%\n", "depth 4 share of the tree's decisions resting on EmployeeID: 3.7%\n", "depth 6 share of the tree's decisions resting on EmployeeID: 2.7%\n", "depth 10 share of the tree's decisions resting on EmployeeID: 3.9%\n", "id_importance_3 = 5.3%\n", "id_importance_10 = 3.9%\n" ] }, { "data": { "text/plain": [ "'3.9%'" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| caption: How much of the tree's decision making rests on the ID column\n", "for d in (3, 4, 6, 10):\n", " imp = report(WITH_ID, d)[\"importances\"].get(\"EmployeeID\", 0.0)\n", " print(f\"depth {d:>2} share of the tree's decisions resting on EmployeeID: {imp:.1%}\")\n", "\n", "record(\"id_importance_3\", f\"{report(WITH_ID, 3)['importances'].get('EmployeeID', 0):.1%}\")\n", "record(\"id_importance_10\", f\"{report(WITH_ID, 10)['importances'].get('EmployeeID', 0):.1%}\")" ] }, { "cell_type": "markdown", "id": "af50d1d8", "metadata": {}, "source": [ "## 3. Why a useless column gets used at all\n", "\n", "Because **a column with more distinct values gets more chances to look good.**\n", "\n", "A yes/no column offers the tree exactly one way to cut it. A column of 975\n", "different numbers offers 974 — and the best of 974 lucky cuts looks better than\n", "the best of one, even when every one of them is meaningless.\n", "\n", "Here is that measured, with columns that are pure noise by construction." ] }, { "cell_type": "code", "execution_count": 4, "id": "7e8b4ec1", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:54.977318Z", "iopub.status.busy": "2026-08-19T13:19:54.977237Z", "iopub.status.idle": "2026-08-19T13:19:55.255987Z", "shell.execute_reply": "2026-08-19T13:19:55.255698Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a meaningless column with 2 distinct values best gain 0.0002\n", "a meaningless column with 4 distinct values best gain 0.0006\n", "a meaningless column with 10 distinct values best gain 0.0008\n", "a meaningless column with 50 distinct values best gain 0.0011\n", "a meaningless column with 200 distinct values best gain 0.0013\n", "a meaningless column with 975 distinct values best gain 0.0013\n", "noise_by_cardinality = [{'distinct': 2, 'mean': 0.0002}, {'distinct': 4, 'mean': 0.0006}, {'distinct': 10, 'mean': 0.0008}, {'distinct': 50, 'mean': 0.0011}, {'distinct': 200, 'mean': 0.0013}, {'distinct': 975, 'mean': 0.0013}]\n", "noise_2 = 0.0002\n", "noise_975 = 0.0013\n", "\n", "the real OverTime best gain 0.0110\n", "\n", "the real StockOptionLevel best gain 0.0023\n", "\n", "the real EmployeeID best gain 0.0007\n", "gain_overtime = 0.011\n", "gain_stock = 0.0023\n", "gain_id = 0.0007\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "figure lucky-columns -> the-column-that-gives-it-away.lucky-columns.{light,dark}.svg\n" ] } ], "source": [ "#| caption: Pure noise, scored — the more distinct values it has, the better it looks\n", "def gini(v):\n", " if len(v) == 0:\n", " return 0.0\n", " p = v.mean()\n", " return 1 - p * p - (1 - p) ** 2\n", "\n", "def best_gain(col, target):\n", " \"\"\"The best a single question about this column could possibly do.\"\"\"\n", " order = np.argsort(col, kind=\"stable\")\n", " c, t = np.asarray(col)[order], np.asarray(target)[order]\n", " n, g0, best = len(t), gini(np.asarray(target)), 0.0\n", " for i in range(1, n):\n", " if c[i] == c[i - 1]:\n", " continue\n", " best = max(best, g0 - (i * gini(t[:i]) + (n - i) * gini(t[i:])) / n)\n", " return best\n", "\n", "rng = np.random.default_rng(11)\n", "noise = []\n", "for k in (2, 4, 10, 50, 200, 975):\n", " gains = [best_gain(rng.integers(0, k, len(tr)), ytr.values) for _ in range(30)]\n", " noise.append({\"distinct\": k, \"mean\": round(float(np.mean(gains)), 4)})\n", " print(f\"a meaningless column with {k:>4} distinct values best gain {np.mean(gains):.4f}\")\n", "\n", "record(\"noise_by_cardinality\", noise)\n", "record(\"noise_2\", noise[0][\"mean\"])\n", "record(\"noise_975\", noise[-1][\"mean\"])\n", "\n", "real = {\n", " \"OverTime\": best_gain((df.OverTime == \"Yes\").astype(int).iloc[tr].values, ytr.values),\n", " \"StockOptionLevel\": best_gain(df.StockOptionLevel.iloc[tr].values, ytr.values),\n", " \"EmployeeID\": best_gain(df.EmployeeID.iloc[tr].values, ytr.values),\n", "}\n", "for k, v in real.items():\n", " print(f\"\\nthe real {k:<18} best gain {v:.4f}\")\n", "record(\"gain_overtime\", round(real[\"OverTime\"], 4))\n", "record(\"gain_stock\", round(real[\"StockOptionLevel\"], 4))\n", "record(\"gain_id\", round(real[\"EmployeeID\"], 4))\n", "\n", "def plot(ax):\n", " ax.plot([n[\"distinct\"] for n in noise], [n[\"mean\"] for n in noise],\n", " marker=\"o\", ms=5, color=\"#9aa0aa\", label=\"a column that is pure noise\")\n", " ax.set_xscale(\"log\")\n", " for label, v, c in [(\"the real OverTime column\", real[\"OverTime\"], \"#e2574c\"),\n", " (\"the real StockOptionLevel column\", real[\"StockOptionLevel\"], \"#3b6fd4\"),\n", " (\"the real EmployeeID column\", real[\"EmployeeID\"], \"#c2871a\")]:\n", " ax.axhline(v, ls=\"--\", lw=1.1, color=c)\n", " ax.text(975, v + 0.0004, label, ha=\"right\", fontsize=8.5, color=c)\n", " ax.set_xlabel(\"Distinct values the column has\")\n", " ax.set_ylabel(\"Best gain a single question can get\")\n", " ax.set_ylim(0, real[\"OverTime\"] * 1.35)\n", " ax.legend(frameon=False, fontsize=9, loc=\"center left\")\n", "\n", "save_fig(\"lucky-columns\", plot, figsize=(7, 4.0))" ] }, { "cell_type": "markdown", "id": "de0f0d9e", "metadata": {}, "source": [ "## 4. The number that should worry you\n", "\n", "Look at where the real `StockOptionLevel` column lands against the noise.\n", "\n", "It is a genuine column, describing a genuine thing, and the best question you\n", "can ask about it scores about what a **meaningless** column with fifty values\n", "gets by luck. The employee number scores less than noise with four.\n", "\n", "**The tree cannot tell those apart.** It has no idea which columns are facts\n", "about a person and which are filing conventions, and it will spend its splits on\n", "either." ] }, { "cell_type": "code", "execution_count": 5, "id": "8a0de604", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:55.257156Z", "iopub.status.busy": "2026-08-19T13:19:55.257067Z", "iopub.status.idle": "2026-08-19T13:19:55.259322Z", "shell.execute_reply": "2026-08-19T13:19:55.259146Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "noise_50 = 0.0011\n", "real StockOptionLevel 0.0023\n", "noise with 50 values 0.0011\n", "real EmployeeID 0.0007\n", "noise with 4 values 0.0006\n" ] } ], "source": [ "#| caption: Real columns and noise on the same scale\n", "record(\"noise_50\", [n[\"mean\"] for n in noise if n[\"distinct\"] == 50][0])\n", "print(f\"real StockOptionLevel {real['StockOptionLevel']:.4f}\")\n", "print(f\"noise with 50 values {[n['mean'] for n in noise if n['distinct'] == 50][0]:.4f}\")\n", "print(f\"real EmployeeID {real['EmployeeID']:.4f}\")\n", "print(f\"noise with 4 values {[n['mean'] for n in noise if n['distinct'] == 4][0]:.4f}\")" ] }, { "cell_type": "markdown", "id": "976be393", "metadata": {}, "source": [ "## 5. Now the dangerous version\n", "\n", "An employee number is a nuisance. A column that already knows the answer is a\n", "catastrophe, and it is far more common.\n", "\n", "**This file does not contain one, so the next cell adds one on purpose.** It\n", "invents `ExitInterviewBooked` — a column any real HR system would hold, true for\n", "almost everyone who left and almost nobody who stayed. It is not a fact about\n", "the person. It is a consequence of the thing we are trying to predict.\n", "\n", "Watch what it does to the score." ] }, { "cell_type": "code", "execution_count": 6, "id": "9eeabde6", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:55.260544Z", "iopub.status.busy": "2026-08-19T13:19:55.260471Z", "iopub.status.idle": "2026-08-19T13:19:55.274667Z", "shell.execute_reply": "2026-08-19T13:19:55.273995Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " on what it saw on strangers leavers found\n", "without the invented column 0.884 0.857 18 of 71\n", "with it 0.960 0.960 56 of 71\n", "clean_train = 88.4%\n", "clean_test = 85.7%\n", "clean_caught = 18\n", "clean_gap = 2.7%\n", "leaky_train = 96.0%\n", "leaky_test = 96.0%\n", "leaky_caught = 56\n", "leaky_gap = 0.0%\n", "leak_importance = 88.5%\n", "n_test_leavers = 71\n" ] }, { "data": { "text/plain": [ "71" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| caption: An invented column that already knows the answer — added deliberately\n", "# CONSTRUCTED, not in the file. True for 90% of leavers and 4% of stayers,\n", "# which is roughly how an exit-interview flag behaves in a real HR system.\n", "rng2 = np.random.default_rng(5)\n", "leak = np.where(y == 1, rng2.random(len(y)) < 0.90, rng2.random(len(y)) < 0.04).astype(int)\n", "\n", "WITH_LEAK = WITH_ID.copy()\n", "WITH_LEAK[\"ExitInterviewBooked\"] = leak\n", "\n", "clean, leaky = report(WITH_ID), report(WITH_LEAK)\n", "print(f\"{'':34} {'on what it saw':>15} {'on strangers':>14} {'leavers found':>15}\")\n", "for label, r in ((\"without the invented column\", clean), (\"with it\", leaky)):\n", " print(f\"{label:<34} {r['train']:>15.3f} {r['test']:>14.3f} {r['caught']:>10} of {int(yte.sum())}\")\n", "\n", "record(\"clean_train\", f\"{clean['train']:.1%}\")\n", "record(\"clean_test\", f\"{clean['test']:.1%}\")\n", "record(\"clean_caught\", clean[\"caught\"])\n", "record(\"clean_gap\", f\"{clean['train'] - clean['test']:.1%}\")\n", "record(\"leaky_train\", f\"{leaky['train']:.1%}\")\n", "record(\"leaky_test\", f\"{leaky['test']:.1%}\")\n", "record(\"leaky_caught\", leaky[\"caught\"])\n", "record(\"leaky_gap\", f\"{abs(leaky['train'] - leaky['test']):.1%}\")\n", "record(\"leak_importance\", f\"{leaky['importances']['ExitInterviewBooked']:.1%}\")\n", "record(\"n_test_leavers\", int(yte.sum()))" ] }, { "cell_type": "markdown", "id": "46291a87", "metadata": {}, "source": [ "## 6. And it leaves no mark\n", "\n", "This is the part worth stopping on.\n", "\n", "Lesson 6's test for a broken model was **the gap** — high on what it learned\n", "from, low on what it has never seen. The leaky model has no gap. It is equally\n", "good on both, and it is far better than anything honest.\n", "\n", "**A held-out set cannot catch this.** Every safeguard from the last lesson is\n", "in place, working correctly, and reporting a triumph." ] }, { "cell_type": "code", "execution_count": 7, "id": "6290545f", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:55.276185Z", "iopub.status.busy": "2026-08-19T13:19:55.276070Z", "iopub.status.idle": "2026-08-19T13:19:55.321880Z", "shell.execute_reply": "2026-08-19T13:19:55.321674Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "figure the-leak-has-no-tell -> the-column-that-gives-it-away.the-leak-has-no-tell.{light,dark}.svg\n" ] } ], "source": [ "#| caption: The gap that catches overfitting, and the leak it does not catch\n", "def plot2(ax):\n", " labels = [\"honest tree\\n(no invented column)\", \"leaky tree\\n(with it)\"]\n", " trains = [clean[\"train\"], leaky[\"train\"]]\n", " tests = [clean[\"test\"], leaky[\"test\"]]\n", " xs = range(2)\n", " ax.bar([x - 0.19 for x in xs], trains, width=0.36, color=\"#9aa0aa\", label=\"on what it saw\")\n", " ax.bar([x + 0.19 for x in xs], tests, width=0.36, color=\"#e2574c\", label=\"on strangers\")\n", " for x, (a, b) in enumerate(zip(trains, tests)):\n", " ax.text(x - 0.19, a + 0.006, f\"{a:.3f}\", ha=\"center\", fontsize=9)\n", " ax.text(x + 0.19, b + 0.006, f\"{b:.3f}\", ha=\"center\", fontsize=9)\n", " ax.text(x, 0.60, f\"gap {abs(a - b):.3f}\", ha=\"center\", fontsize=9.5, color=\"#77777f\")\n", " ax.set_xticks(list(xs))\n", " ax.set_xticklabels(labels, fontsize=9)\n", " ax.set_ylim(0.55, 1.02)\n", " ax.set_ylabel(\"Share called right\")\n", " ax.legend(frameon=False, fontsize=9, loc=\"upper left\")\n", " ax.grid(axis=\"x\", visible=False)\n", "\n", "save_fig(\"the-leak-has-no-tell\", plot2, figsize=(7, 3.8))" ] }, { "cell_type": "markdown", "id": "4fe8288d", "metadata": {}, "source": [ "## 7. What does catch it\n", "\n", "Read the model.\n", "\n", "Ask which columns it is actually leaning on, and one of them is holding up\n", "nearly the whole thing. Then say the column's name out loud to anybody who works\n", "in HR, and you are finished in one sentence: *we only book an exit interview\n", "after somebody has resigned.*\n", "\n", "That sentence is not in the data. It is not available to any amount of\n", "cross-validation. **It is available to a person who can read the model**, which\n", "is the entire reason this path started with a tree." ] }, { "cell_type": "code", "execution_count": 8, "id": "3036c0e1", "metadata": { "execution": { "iopub.execute_input": "2026-08-19T13:19:55.322950Z", "iopub.status.busy": "2026-08-19T13:19:55.322862Z", "iopub.status.idle": "2026-08-19T13:19:55.329844Z", "shell.execute_reply": "2026-08-19T13:19:55.329573Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ExitInterviewBooked 88.5% ############################################\n", "YearsAtCompany 3.8% ##\n", "OverTime_Yes 2.8% #\n", "JobSatisfaction 1.2% #\n", "TotalWorkingYears 1.0% #\n", "leak_top_columns = [{'column': 'ExitInterviewBooked', 'share': '88.5%'}, {'column': 'YearsAtCompany', 'share': '3.8%'}, {'column': 'OverTime_Yes', 'share': '2.8%'}, {'column': 'JobSatisfaction', 'share': '1.2%'}, {'column': 'TotalWorkingYears', 'share': '1.0%'}]\n", "leaky_tree = {'q': 'Exit interview booked?', 'note': '975 people - 131 left', 'yes': {'q': 'Works overtime?', 'note': '152 people - 119 left', 'yes': {'leaf': 'leaves', 'note': '76 people - 69 left'}, 'no': {'leaf': 'leaves', 'note': '76 people - 50 left'}}, 'no': {'q': 'Distance from home under 24?', 'note': '823 people - 12 left', 'yes': {'leaf': 'stays', 'note': '784 people - 9 left'}, 'no': {'leaf': 'stays', 'note': '39 people - 3 left'}}}\n" ] }, { "data": { "text/plain": [ "{'q': 'Exit interview booked?',\n", " 'note': '975 people - 131 left',\n", " 'yes': {'q': 'Works overtime?',\n", " 'note': '152 people - 119 left',\n", " 'yes': {'leaf': 'leaves', 'note': '76 people - 69 left'},\n", " 'no': {'leaf': 'leaves', 'note': '76 people - 50 left'}},\n", " 'no': {'q': 'Distance from home under 24?',\n", " 'note': '823 people - 12 left',\n", " 'yes': {'leaf': 'stays', 'note': '784 people - 9 left'},\n", " 'no': {'leaf': 'stays', 'note': '39 people - 3 left'}}}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#| caption: Which columns the leaky tree is leaning on\n", "top = [(c, v) for c, v in leaky[\"importances\"].items() if v > 0][:5]\n", "for c, v in top:\n", " print(f\"{c:<24} {v:>6.1%} {'#' * round(v * 50)}\")\n", "\n", "record(\"leak_top_columns\", [{\"column\": c, \"share\": f\"{v:.1%}\"} for c, v in top])\n", "\n", "LABEL = {\"ExitInterviewBooked\": \"Exit interview booked?\", \"OverTime_Yes\": \"Works overtime?\",\n", " \"JobSatisfaction\": \"Job satisfaction\", \"YearsAtCompany\": \"Years at company\",\n", " \"MonthlyIncome\": \"Monthly income\", \"Age\": \"Age\", \"EmployeeID\": \"Employee number\",\n", " \"StockOptionLevel\": \"Stock option level\", \"TotalWorkingYears\": \"Total working years\",\n", " \"DistanceFromHome\": \"Distance from home\", \"Education\": \"Education level\"}\n", "\n", "def to_nodes(clf, names, node=0):\n", " \"\"\"sklearn's arrays -> the shape the lesson draws. sklearn sends\n", " `feature <= threshold` LEFT, so a yes/no column has its NO side on the\n", " left and the branches are swapped to ask the question the human way.\"\"\"\n", " t = clf.tree_\n", " stayed, gone = t.value[node][0]\n", " n = int(t.n_node_samples[node])\n", " note = f\"{n} people - {round(gone / (stayed + gone) * n)} left\"\n", " if t.children_left[node] == -1:\n", " return {\"leaf\": \"leaves\" if gone >= stayed else \"stays\", \"note\": note}\n", " col, thr = names[t.feature[node]], t.threshold[node]\n", " lo = to_nodes(clf, names, t.children_left[node])\n", " hi = to_nodes(clf, names, t.children_right[node])\n", " if set(np.unique(WITH_LEAK[col])) <= {0, 1, True, False}:\n", " return {\"q\": LABEL.get(col, col), \"note\": note, \"yes\": hi, \"no\": lo}\n", " whole = bool(pd.Series(WITH_LEAK[col]).dropna().mod(1).eq(0).all())\n", " step = np.ceil(thr) if whole else thr\n", " return {\"q\": f\"{LABEL.get(col, col)} under {step:,.0f}?\", \"note\": note, \"yes\": lo, \"no\": hi}\n", "\n", "record(\"leaky_tree\", to_nodes(fit(WITH_LEAK, 2), list(WITH_LEAK.columns)))" ] }, { "cell_type": "markdown", "id": "2b614d1e", "metadata": {}, "source": [ "## Try this\n", "\n", "1. **Weaken the leak.** Make the invented column true for 60% of leavers instead\n", " of 90%. At what strength does it stop being obvious in the importances — and\n", " is it still lifting the score when it stops being visible?\n", "2. **Leak something subtler.** Replace it with a column that is true for anyone\n", " whose `YearsAtCompany` is 0 — a plausible \"new joiner\" flag. Is that a leak?\n", " The answer is not in the data.\n", "3. **Drop the ID and re-run section 2.** Nothing else in the tree changes much,\n", " which is the honest scale of that particular problem: an employee number is a\n", " nuisance rather than a disaster. The invented column is the disaster." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 5 }