Bootstrap Aggregation
Bootstrap aggregation fits several models on resampled training data and combines their predictions. It can reduce variance for suitable unstable base estimators.
Understand the idea
A bagging classifier resamples observations with replacement. The base estimator, number of estimators, and random seed affect its behaviour. Correlated or biased training data remains a problem even when many models are averaged. The small example demonstrates the wrapper API rather than proving an ensemble beats a well-tuned simpler model.
Where to try it
Run locally with scikit-learn and NumPy installed in your virtual environment (py -m pip install scikit-learn numpy). These examples are executed during course review, but the online runner does not currently preload scikit-learn. The small data is for learning the API, not evidence of production model quality.
A worked example
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
X = [[value] for value in range(12)]
y = [0] * 6 + [1] * 6
model = BaggingClassifier(estimator=DecisionTreeClassifier(max_depth=2), n_estimators=8, random_state=7, n_jobs=1)
model.fit(X, y)
print(model.predict([[0], [11]]).tolist())
Read the result
The extreme inputs are predicted as classes 0 and 1 in this constructed example.
Try one change
Compare a base tree and its bagged version on the same held-out data and report uncertainty, not only one score.
For package options, see the scikit-learn documentation.