For my undergraduate thesis at Ege University (Department of Mathematics, supported by TÜBİTAK's 2209-B industry research program), I built a pipeline to predict marketing campaign conversions. The dataset: 48,000 synthetic customer records shaped like real digital-marketing data. Only 980 of those customers converted — a 2.04% positive rate.
My first trained model scored a ROC-AUC of 1.0000. F1: also 1.0000.
For about a minute, that felt great. Then it felt wrong. A perfect score on marketing data — where human behavior is noisy, campaigns are blunt instruments, and 98% of customers ignore you — is not a good result. It is physically impossible. The model wasn't predicting anything. It was reading the answer key.
What the model was actually doing
The feature set included variables like ConversionRate and CTR_to_Conversion. Look at those names for a second. If a feature encodes the rate at which a customer converts, then the label — "did this customer convert?" — is already baked into the input. This is target leakage: features derived from the very outcome you're trying to predict, information the model would never have at inference time, before the conversion has happened.
The insidious part is that leakage doesn't announce itself. The pipeline runs, the metrics come out, and the metrics are fantastic. Everything looks like success. The only symptom is that it looks like too much success.
Hunting the leak: a six-scenario ablation
I couldn't just guess which features were leaking, so I ran a systematic ablation — six scenarios, each removing a suspected group of features and retraining.
The first surprise: removing the two obvious suspects, ConversionRate and CTR_to_Conversion, changed nothing. ROC-AUC stayed at 1.0000. Removing ROI_Proxy on top of them — still 1.0000. That was proof that at least one more feature was carrying the label.
The culprit turned out to be CPA_Proxy, a cost-per-acquisition proxy. On its own, it was enough to sustain the entire perfect score. The moment it left the feature set, ROC-AUC dropped from 1.0000 to 0.7071.
That single number drop is the most honest moment in the whole thesis. Nothing about the model improved or degraded — I just stopped letting it cheat.
The fix: features that exist before the answer does
Deleting leaking features isn't enough if they carried genuinely useful business signal, so I rebuilt them from inputs that exist before a conversion happens:
ROI_v2= (revenue × CTR) / ad spendCPA_v2= ad spend / website visits
The test for every engineered feature became a simple question: could I compute this for a customer who hasn't converted yet? If the answer is no, the feature doesn't go in.
Why 0.71 is the honest number
The final production model — logistic regression with interaction features — scored ROC-AUC 0.713. I also trained XGBoost (0.705) and Random Forest (0.698); logistic regression won not just on the metric but because the downstream system needed interpretable coefficients.
A drop from 1.0 to 0.71 looks like failure if you frame it as "the model got worse." It didn't. The 0.71 is what a model can genuinely know about a customer before they convert. The 1.0 was memorization; the 0.71 is prediction. Those are different activities, and only one of them is useful in production.
A side lesson: SMOTE made things worse
With a 2% positive rate, the textbook says: oversample the minority class. So I tested SMOTE. It degraded every metric — ROC-AUC went from 0.7072 to 0.6963, F1 from 0.0760 to 0.0740. At this level of imbalance, the synthetic minority samples added noise, not signal. I dropped it.
I mention this because "apply SMOTE to imbalanced data" is one of those reflexes that rarely gets checked. It failed here, and the honest move was to say so and move on rather than keep it because the textbook approves.
Living with a "weak" F1
The honest model's F1 was 0.15. Out of context that looks embarrassing, so let me give the context.
At a 2% positive rate, the default 0.5 classification threshold is useless — the model almost never emits probabilities that high for anyone. F1-maximizing threshold optimization landed the operating points around 0.797–0.812. At the 0.797 threshold, the confusion matrix shows a deliberately conservative model: 35 of 196 true converters caught, at the cost of only 227 false positives out of 9,404 negatives. That's a defensible trade when each false alarm has a real marketing cost.
And here's the part that actually justifies the model: prediction wasn't the end goal. The logistic regression fed a prescriptive recommender that, for each customer in the 9,600-customer test set, simulated all 49 channel-platform combinations (7 channels × 7 platforms) and picked the one with the highest predicted conversion probability.
The result: average predicted conversion probability rose from 0.0855 under the existing campaign assignments to 0.1082 under the recommended ones — an average lift of +0.0227, or roughly a 26.5% relative improvement.
One qualification I insist on every time I state that number: this is a simulated, predicted relative lift on the test set. It is what the model believes would happen if customers were reassigned to their optimal channel-platform combination. It is not a live A/B test result, and I won't present it as one. (Also honest: about 15–20% of customers were already on their optimal combination — the recommender correctly gives them near-zero lift instead of inventing an improvement.)
So the chain is: an "ugly" F1 of 0.15, a modest AUC of 0.71 — and a recommender built on top of them that produces a meaningful predicted improvement. Metrics only mean something relative to the decision they power.
The takeaway
If I had shipped the first model, the leak would have surfaced in the worst possible way: perfect offline metrics, useless live predictions, and an audience that had already been promised magic.
What I actually learned:
- A perfect score is a bug report. On real-world-shaped data, ROC-AUC 1.0 means your pipeline is broken, not that you're brilliant.
- Ablate systematically, not intuitively. My intuition flagged two features; the ablation found a third that was doing all the damage alone.
- Leakage checks belong in the pipeline, not the post-mortem. The question "would this feature exist at inference time?" is cheap to ask and catastrophic to skip.
- Report the honest number with pride. 0.71 with a clean feature set is a result. 1.0 with leakage is a liability wearing a medal.
I was wrong for exactly as long as it took to distrust my own good news. That reflex — suspicion first, celebration later — is the most transferable thing this project taught me.