Impact story · Modelling → Strategy
The most valuable thing was what I threw away
My prediction model scored a suspicious 100% — and the smartest move was deleting the data that made it perfect.
A model that is never wrong has usually been told the answer. The interesting part of this project was not building it — it was working out why it was too good.
The situation
A model to prioritise the leads worth chasing
Counsellor time is the scarce resource, not leads. I built a model to predict which leads would convert, so the team could spend that time on the ones most likely to become clients.
The red flag
It scored 100% — which meant it was cheating
A perfect score is not a win, it is a symptom. The model had access to information that only exists after a lead has already converted, so it was reading the answer off the back of the page. Ship that, and the team trusts something that collapses on the first real lead.
What I did
Found the leakage, deleted it, rebuilt honestly
I audited every feature for whether it would actually be available at scoring time, removed the leaking fields, and retrained. The honest model lands at 77% accuracy against a 62% majority-class baseline — a real 15-point lift over guessing.
The commercial result
A defensible way to rank leads and channels
The top-scoring 20% of leads convert at 84%, more than double the average. And the model surfaced something leadership could act on immediately: referral leads convert at 90% against 26% for paid social. That is a quality-over-volume argument with evidence behind it.
Accuracy: the leaking model, the honest model, and the baseline
Majority-class baseline is what you get by guessing 'will not convert' every time
View as table
| Model | Accuracy |
|---|---|
| Leaking model | 100% |
| Honest model | 77% |
| Baseline (guess) | 62% |
The bar on top is the one to distrust. The only number that means anything here is the gap between the honest model and the baseline: 77% against 62%. That gap is the actual value the model adds over doing nothing, and it is the number I would defend in a room.
What the model was reading
Leakage is rarely malicious and almost never obvious. It hides in fields that look innocuous until you ask when does this value get written? — the question that separates a feature from a label in disguise.
| Field | Why it leaked | Verdict |
|---|---|---|
| Pipeline stage | Only advances to the later stages once the lead has already converted. | Removed |
| Payment / invoice fields | Populated at the moment of conversion — the label wearing a different name. | Removed |
| Total touchpoint count | Counted the full lifetime of the record, including activity after conversion. | Rebuilt |
| Source / channel | Known the moment the lead arrives. Genuinely available at scoring time. | Kept |
| First-response time | Known early in the lifecycle, before any conversion decision. | Kept |
The fix for "total touchpoints" is worth calling out, because deleting it would have thrown away real signal. Instead I rebuilt it as a point-in-time feature — touchpoints within the first 72 hours — which is knowable at scoring time and kept most of its predictive power.
From model to decision
An accuracy figure is not an insight. What made this useful to leadership was translating it into two things they could act on: which leads to work first, and which channels are worth the money.
Conversion rate by lead score decile
Leads ranked by model score, split into ten equal groups
View as table
| Score decile | Conversion rate |
|---|---|
| Top 10% | 89% |
| 10–20% | 79% |
| 20–40% | 52% |
| 40–60% | 33% |
| 60–80% | 17% |
| Bottom 20% | 6% |
Illustrative deciles; the top-20% and overall-average figures are the measured ones.
The top two deciles — the highest-scoring 20% of leads — convert at 84%, more than double the overall average. That is a concrete working instruction: these are the calls to make first thing in the morning.
Conversion rate by acquisition channel
The same leads, cut by where they came from
View as table
| Channel | Conversion rate |
|---|---|
| Referral | 90% |
| Organic / SEO | 48% |
| Paid social | 26% |
Referral leads convert at 90% against 26% for paid social — a 3.5× difference in quality that a cost-per-lead figure completely hides. A cheap lead that never converts is not cheap. This gave the team a defensible reason to shift effort toward the sources that actually produce clients.
Why I tell this story
It shows the two things I want a team to know about how I work. I am technically capable enough to build the thing — and I care more about a number being true than impressive. A 100% model would have looked better in a slide deck for exactly as long as it took someone to rely on it.
And the output was not a model. It was a commercial insight about consumer behaviour — quality over volume, with the evidence to back it — which is where analytics actually earns its keep.
← All projects