bhylu.com

Advertisements

AI Data Leakage

Fit preprocessing statistics on training data only.

How it works

Leakage happens when a model or a modelling decision uses information unavailable in its intended evaluation setting. Preprocessing can leak even without target labels: a mean computed from held-out features influences the transformation. Keep fitted preprocessing inside a training pipeline and apply its learned parameters to later records.

Run the code

Try in playground loads this complete standard-library Python example without executing it. Choose Run when ready. Local review uses Python 3.12.14; the existing browser editor uses Pyodide 0.27.7. These small fictional examples need no account, external data, model-provider key or database service.

Download the complete data-leakage.py

Example

data-leakage.py

from statistics import mean

train = [2.0, 4.0, 6.0]
test = [100.0]
training_mean = mean(train)
print("training mean:", training_mean)
print("transformed held-out:", [v-training_mean for v in test])
print("leaking combined mean:", mean(train+test))

Expected result

training mean: 4.0
transformed held-out: [96.0]
leaking combined mean: 28.0

The held-out value changes the combined mean from four to twenty-eight. That is precisely why it must not help fit the training transformation. An extreme future value should remain visibly extreme rather than making the training data look more centred.

Technical references