Data leakage in ML: find it before you ship

What is data leakage in machine learning?

Data leakage happens when model development uses information that would not be available for the prediction being evaluated. The resulting score can make a weak model appear useful. Scikit-learn's common-pitfalls guide warns against fitting preprocessing on test data and recommends keeping fitted transformations inside the training process. This matters whether you wrote the notebook yourself or used an AI coding assistant: a successful notebook run does not validate the experiment. The delivery-delay scenario below is a proposed audit exercise, with no claimed model results.

• Source: scikit-learn on data leakage

Define when the prediction happens

Imagine predicting whether a fictional parcel will arrive late at the moment it is dispatched. Write that prediction time above your feature list. Information recorded after dispatch, such as final delivery status or actual arrival time, cannot be an input. Build a small data dictionary with column name, meaning, event time, time the value became available, and whether the production system could read it at dispatch. This audit can catch a misleading feature before you spend time comparing algorithms.

Choose a split that matches the intended use

A random row split can be unsuitable when records share an entity or depend on time. Scikit-learn provides group-aware splitters for holding out groups and TimeSeriesSplit for ordered evaluation. For the parcel exercise, use earlier dispatch periods to develop the model and a later period to evaluate future performance. If the goal is performance on entirely new customers, hold out customers too. Sorting rows is not enough: ensure that every training label was actually available by the evaluation cutoff. A splitter cannot repair future information already embedded in a feature.

• Source: scikit-learn cross-validation guide

Fit preprocessing only inside the training boundary

Learn imputation values, scaling, feature selection, and other fitted transformations from training data. Apply those fitted transformations to validation or test data. During cross-validation, fit them again inside each training fold; preprocessing the full dataset before cross-validation breaks that separation. A scikit-learn Pipeline can keep preprocessing and the estimator together, but it cannot know that a column contains future information. Inspect feature provenance and splitting as well as the code structure.

• Source: keeping preprocessing separate from test data

Run a controlled leakage audit

Use a permitted dataset with documented timestamps, or label your generated parcel records as synthetic. Keep a clearly marked diagnostic experiment separate from the model you present as usable. Start by recording the current features, split, and metric. Remove post-dispatch information, rebuild time-dependent features using only eligible history, and rerun using the intended split. Hold other settings constant where possible. A score drop is a finding to explain. It is not something to hide by reporting only the original result.

• Check duplicate parcel IDs before splitting. Repeated copies of the same event must not create an accidental shortcut across evaluation boundaries.
• Inspect the strongest-looking features manually. A status code or filename can encode the outcome even when its name sounds harmless.
• Keep a manifest of the selected training and evaluation records, with permitted identifiers or hashes, so the partition can be reproduced.
• Log every experiment you use to make a choice. Repeatedly tuning against the final test results turns that set into development data.

Compare with a baseline and the cost of errors

For this exercise, compare the model with a simple rule that always predicts the most common outcome, learned from the training set. If late deliveries are rare, accuracy can be high while missing the very cases you want to catch. Show a confusion matrix and report precision and recall for the late class, alongside the sample counts. Explain the decision the prediction would support: how many parcels could a hypothetical operations team investigate, and what happens when an alert is wrong? Do not describe simulated results as delivered business savings.

• Source: scikit-learn classification metrics

What should a reviewer be able to reproduce?

Publish the permitted data or generation method, data dictionary, prediction-time rule, split method, dependency versions, preprocessing pipeline, baseline, and final evaluation. Include the audit findings and a command or notebook sequence that reproduces the report from a clean start. Document missing timestamps, small sample sizes, synthetic assumptions, and populations you did not evaluate. A defensible project can have modest scores. Its value is that the reader can understand what those scores measure and why they should trust the process.

• Use the portfolio case-study template