Module · Decision Trees✅Knowledge checkCheck: scores that are not what they look like5 questions · Pass at 60%1. 86.5% of the employees stayed. A model scores 86.5% on held-out data. What has it earned?A strong result — nearly nine in ten correctNothing yet: that is what saying 'stays' to everybody getsIt depends on the depth of the treeIt has earned the right to be deployed2. One leaf of the accurate shallow tree holds 107 people, 43 of whom left, and calls them stayers. Why?The tree was fitted incorrectlyThe leaf was too small to make a reliable callStayers outnumber leavers in that leaf, so 'stays' maximises accuracy — the only thing it was asked to maximise40% is below the threshold for statistical significance3. Adding an invented 'exit interview booked' column took the tree to 96.0% on people it learned from and 96.0% on strangers. What is the tell?The gap between the two scoresThe unusually high training scoreThere is no tell in those numbers — the gap is zero, which is exactly why a held-out set cannot catch itThe score is suspiciously round4. A column of pure noise with 975 distinct values scores six times what pure noise with two values scores. Why?More values means more informationMore distinct values means more thresholds to try, so the best of them looks better by luckGini impurity is undefined for wide columnsThe extra values correlate with the outcome5. What actually caught the leaking column?Cross-validation with more foldsA larger held-out setReading the model: one column held 88.5% of the tree, and its name gave it away in a sentenceComparing gini against entropyCheck answers← Back to Decision TreesNext topic →
