Data Science Missing Data
Count missing measurements before choosing how to handle them.
How it works
Missing values can arise from equipment failure, non-response or a field that does not apply. Their cause matters because the observed rows may no longer represent the intended population. Deleting rows changes coverage; imputing a value adds an assumption. Pandas distinguishes missing values from ordinary zero. First measure the missingness, then choose a policy suited to the question and evaluation design.
Run the code
Try in playground opens the correct pandas editor and preloads the complete source. Choose Run to execute; loading does not execute it. The isolated browser runtime does not read personal PC files.
Local review: Python 3.12.14, NumPy 2.5.4, Pandas 3.0.6 and SciPy 1.18.0. Browser review: Pyodide 0.27.7, Python 3.12.7, NumPy 2.0.2, Pandas 2.2.3 and SciPy 1.14.1 in their respective editors. Save the download as missing-data.py and run python missing-data.py in your separate environment. Apache and MySQL are not needed. Environment setup.
Example
import pandas as pd
litres = pd.Series([18., None, 22.])
assert litres.isna().sum() == 1
observed = litres.dropna()
assert observed.mean() == 20.
print(f"Observed mean {observed.mean():.1f} L from {observed.size}/3 days")
Expected result
Observed mean 20.0 L from 2/3 days
The example reports both the observed mean and its coverage. It does not pretend the missing day dispensed no water. If the meter fails more often on very busy days, the observed mean could be biased. A modelling pipeline must learn an imputation rule from training data only and apply the stored rule to validation data.
Technical references: NumPy, Pandas and SciPy. For evaluation design see the scikit-learn guide to leakage; these examples do not require scikit-learn.