.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/pitfalls_and_solutions/plot_skd001_skd002_overfitting_underfitting.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_pitfalls_and_solutions_plot_skd001_skd002_overfitting_underfitting.py: .. _example_skd001_skd002_overfitting_underfitting: SKD001 & SKD002 - Overfitting and underfitting ============================================== :ref:`SKD002 ` and :ref:`SKD001 ` describe opposite ends of the same problem: model *expressiveness*. - Too little expressiveness: train and test scores are close to those of a model that guesses randomly (often called a dummy model) (SKD002). - Too much expressiveness: train scores pull far ahead of test scores (SKD001). A performant model learns from the training data without memorizing its specificities, so that learned knowledge can be generalized to new data. This notebook walks the underfitting to overfitting path while showing different mitigations techniques: - tweaking model capacity (model family / complexity) - feature engineering - tuning regularization - using early stopping - using more data We use California housing (median house value). Half of the rows feed the main walkthrough; the other half is reserved to show the effect of adding training data. .. GENERATED FROM PYTHON SOURCE LINES 32-41 Load the California housing dataset =================================== Each row is a census block group in a geographical region whose coordinates we have (``Latitude``, ``Longitude``). The features mix house characteristics (age, rooms, occupancy) with regional aggregates (median income, population). The target ``MedHouseVal`` is the median house value for the block group. The feature set is small but rich: later we combine columns such as ``AveRooms / AveOccup`` into a rooms-per-person marker. .. GENERATED FROM PYTHON SOURCE LINES 41-55 .. code-block:: Python import pandas as pd from sklearn.datasets import fetch_california_housing from sklearn.model_selection import train_test_split housing = fetch_california_housing(as_frame=True) X, X_heldout, y, y_heldout = train_test_split( housing.data, housing.target, train_size=0.5, random_state=42, ) .. GENERATED FROM PYTHON SOURCE LINES 56-58 Let us glance at the table. Pay attention to ``AveRooms`` and ``AveOccup``: we will combine them into a rooms-per-person feature below. .. GENERATED FROM PYTHON SOURCE LINES 58-63 .. code-block:: Python from skrub import TableReport TableReport(X) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 64-66 ``MedHouseVal`` is capped at 5.0 (500k USD). Click the column in the report to see the histogram and the pile-up at the ceiling. .. GENERATED FROM PYTHON SOURCE LINES 66-69 .. code-block:: Python TableReport(y) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 70-72 We use the same split for every model comparison, with a fixed seed, so that we are sure that we are comparing models on the same test data. .. GENERATED FROM PYTHON SOURCE LINES 72-77 .. code-block:: Python from skore import TrainTestSplit splitter = TrainTestSplit(test_size=0.2, random_state=42) .. GENERATED FROM PYTHON SOURCE LINES 78-84 Underfitting: SKD002 fires ========================== We start with a linear model that is overly regularized. :class:`~sklearn.linear_model.Ridge` with a very large ``alpha`` shrinks coefficients toward zero, so predictions stay close to a dummy baseline. .. GENERATED FROM PYTHON SOURCE LINES 84-96 .. code-block:: Python import skore from sklearn.linear_model import Ridge report_underfit = skore.evaluate( Ridge(alpha=1e6), X=X, y=y, splitter=splitter, ) report_underfit.metrics.summarize(data_source="both").frame() .. raw:: html
Ridge (train) Ridge (test)
metric
r2 0.041240 0.040223
rmse 1.127863 1.150918
mae 0.889223 0.907935
mape 0.607538 0.614893
fit_time 0.002820 0.002820
predict_time 0.000999 0.001038


.. GENERATED FROM PYTHON SOURCE LINES 97-98 For a model that learns so little, SKD002 fires. .. GENERATED FROM PYTHON SOURCE LINES 98-101 .. code-block:: Python report_underfit.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 102-109 Increase model expressiveness ============================= Moving away from underfitting means giving the model enough expressiveness to use the inputs. Let us drop to a :class:`~sklearn.linear_model.Ridge` with a much smaller ``alpha``: it already learns useful weights on each feature and is enough to clear SKD002 on this table. .. GENERATED FROM PYTHON SOURCE LINES 109-113 .. code-block:: Python report_ridge = skore.evaluate(Ridge(alpha=1.0), X=X, y=y, splitter=splitter) report_ridge.metrics.summarize(data_source="both").frame() .. raw:: html
Ridge (train) Ridge (test)
metric
r2 0.610830 0.609422
rmse 0.718573 0.734197
mae 0.525393 0.539913
mape 0.313230 0.320523
fit_time 0.002704 0.002704
predict_time 0.000653 0.000795


.. GENERATED FROM PYTHON SOURCE LINES 114-116 .. code-block:: Python report_ridge.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 117-130 Feature engineering =================== Better features help when the model family is fine but the representation is weak (still an underfitting problem). Here, ``AveRooms / AveOccup`` is a rooms-per-person signal that a linear model can use more easily than the raw counts. The same step has an overfitting side effect: every new feature also increases expressiveness. Mild engineering can close an underfit gap; aggressive expansions (very high-degree interactions, huge one-hot spaces) can reopen an overfit gap. Think of features as capacity you add to the inputs, not only to the estimator. .. GENERATED FROM PYTHON SOURCE LINES 130-136 .. code-block:: Python X_fe = X.assign(RoomsPerPerson=X["AveRooms"] / X["AveOccup"].clip(lower=0.1)) report_ridge_fe = skore.evaluate(Ridge(alpha=1.0), X=X_fe, y=y, splitter=splitter) report_ridge_fe.metrics.summarize(data_source="both").frame() .. raw:: html
Ridge (train) Ridge (test)
metric
r2 0.640718 0.639733
rmse 0.690429 0.705134
mae 0.494856 0.508096
mape 0.295383 0.301925
fit_time 0.002891 0.002891
predict_time 0.001097 0.001024


.. GENERATED FROM PYTHON SOURCE LINES 137-139 The test score rises while the train score stays close: we reduced underfitting without creating overfitting by adding a signal-rich feature. .. GENERATED FROM PYTHON SOURCE LINES 139-142 .. code-block:: Python report_ridge_fe.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 143-152 Overfitting: SKD001 fires ========================= Hand-crafted features are one way to capture nonlinearity for a linear model. Another is to switch to a model family that learns nonlinear structure on its own. A default :class:`~sklearn.ensemble.RandomForestRegressor` does that, but unrestricted leaves can also memorize the training data: train metrics look excellent, test metrics improve but not as much as train ones, and SKD001 flags the gap. .. GENERATED FROM PYTHON SOURCE LINES 152-163 .. code-block:: Python from sklearn.ensemble import RandomForestRegressor report_rf = skore.evaluate( RandomForestRegressor(random_state=42), X=X, y=y, splitter=splitter, ) report_rf.metrics.summarize(data_source="both").frame() .. raw:: html
RandomForestRegressor (train) RandomForestRegressor (test)
metric
r2 0.969057 0.784944
rmse 0.202621 0.544797
mae 0.134065 0.361575
mape 0.077009 0.206075
fit_time 5.249770 5.249770
predict_time 0.181115 0.070329


.. GENERATED FROM PYTHON SOURCE LINES 164-166 .. code-block:: Python report_rf.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 167-181 Regularization ============== Once SKD001 appears, we pull expressiveness back with capacity limits on the estimator (leaf size, feature fraction, depth, learning rate, …). Here, we set the hyperparameters of the model by hand. In practice, it is rather hard to know in advance which combination of hyperparameters leads to generalization. One should lean towards tuning hyperparameters. We would advocate for randomized search, with successive halving when the amount of samples is large. See `tuning the hyper-parameters of an estimator `_. In this particular example, we are not implementing the search in order to keep the execution time of the example short. .. GENERATED FROM PYTHON SOURCE LINES 181-195 .. code-block:: Python model_rf_reg = RandomForestRegressor( min_samples_leaf=20, max_features=0.5, random_state=42, ) report_rf_reg = skore.evaluate( model_rf_reg, X=X, y=y, splitter=splitter, ) report_rf_reg.metrics.summarize(data_source="both").frame() .. raw:: html
RandomForestRegressor (train) RandomForestRegressor (test)
metric
r2 0.809771 0.762550
rmse 0.502389 0.572461
mae 0.344942 0.394609
mape 0.200611 0.226745
fit_time 1.802544 1.802544
predict_time 0.080326 0.027588


.. GENERATED FROM PYTHON SOURCE LINES 196-198 .. code-block:: Python report_rf_reg.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 199-219 Early stopping ============== For iterative learners, early stopping is another way to limit expressiveness that does not require a hyperparameter search. We hold out a validation set, monitor a metric, and stop when that metric stops improving. Further iterations are assumed to overfit. We can either pass a validation set to ``.fit`` or use the ``validation_fraction`` parameter. See scikit-learn's example on `gradient boosting with early stopping `_. Let us compare a :class:`~sklearn.ensemble.HistGradientBoostingRegressor` with and without early stopping on the same split to see the effect. We bump ``max_iter`` so that early stopping has room to act. With ``validation_fraction``, the early-stopped model trains on only 90% of each train fold; a small test drop relative to the unstopped run can come from that, rather than from stopping itself. Passing an external validation set (for example ``X_heldout`` / ``y_heldout``) would avoid that, but wiring fit parameters through skore currently needs a skrub DataOp, so we keep ``validation_fraction`` here. .. GENERATED FROM PYTHON SOURCE LINES 219-250 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingRegressor report_hgbr = skore.evaluate( HistGradientBoostingRegressor(max_iter=1000, random_state=42), X=X, y=y, splitter=splitter, ) model_hgbr_es = HistGradientBoostingRegressor( early_stopping=True, validation_fraction=0.1, n_iter_no_change=10, max_iter=1000, random_state=42, ) report_hgbr_es = skore.evaluate( model_hgbr_es, X=X, y=y, splitter=splitter, ) skore.compare( { "hgbr": report_hgbr, "hgbr_early_stopping": report_hgbr_es, } ).metrics.summarize(data_source="both").frame() .. raw:: html
hgbr (train) hgbr (test) hgbr_early_stopping (train) hgbr_early_stopping (test)
metric
r2 0.990605 0.831857 0.935434 0.826642
rmse 0.111650 0.481724 0.292687 0.489138
mae 0.078763 0.313363 0.199150 0.316511
mape 0.046624 0.175758 0.114104 0.177941
fit_time 2.507134 2.507134 0.713126 0.713126
predict_time 0.439940 0.128328 0.125657 0.036498


.. GENERATED FROM PYTHON SOURCE LINES 251-253 .. code-block:: Python report_hgbr.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 254-256 .. code-block:: Python report_hgbr_es.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 257-267 More training data ================== Extra labeled rows help both regimes, but they are especially useful against overfitting: the model has fewer opportunities to memorize a small sample. Let us keep one fixed test fold, fit on the original train fold, then refit after concatenating the held-out half of the dataset. We use :func:`~skore.evaluate` with ``splitter="prefit"`` once the estimator is fitted. A side-effect is that fit time is unavailable (skore did not time ``.fit``), so that metric appears as NaN in the reported tables. .. GENERATED FROM PYTHON SOURCE LINES 267-292 .. code-block:: Python X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) model_less = HistGradientBoostingRegressor( early_stopping=True, validation_fraction=0.1, n_iter_no_change=10, random_state=42, ).fit(X_train, y_train) report_less = skore.evaluate(model_less, X_test, y_test, splitter="prefit") model_more = HistGradientBoostingRegressor( early_stopping=True, validation_fraction=0.1, n_iter_no_change=10, random_state=42, ).fit(pd.concat([X_train, X_heldout]), pd.concat([y_train, y_heldout])) report_more = skore.evaluate(model_more, X_test, y_test, splitter="prefit") skore.compare( {"less_training_data": report_less, "more_training_data": report_more} ).metrics.summarize(data_source="test").frame() .. raw:: html
estimator less_training_data more_training_data
metric
r2 0.820257 0.833391
rmse 0.498065 0.479521
mae 0.327594 0.313229
mape 0.185737 0.176323
predict_time 0.016416 0.015607


.. GENERATED FROM PYTHON SOURCE LINES 293-298 In production, a `learning curve `_ (many refits on growing subsets) shows whether more labels are still worth the cost. skore does not wrap that API yet; we can treat it as a scikit-learn-side diagnostic next to these checks. .. GENERATED FROM PYTHON SOURCE LINES 300-305 Summary comparison ================== Here is a side-by-side comparison of the impact of the different techniques we applied, measured on the same test set. .. GENERATED FROM PYTHON SOURCE LINES 305-319 .. code-block:: Python skore.compare( { "ridge_large_alpha": report_underfit, "ridge": report_ridge, "ridge_feature_engineering": report_ridge_fe, "default_rf": report_rf, "regularized_rf": report_rf_reg, "hgbr": report_hgbr, "hgbr_early_stopping": report_hgbr_es, "more_data": report_more, } ).metrics.summarize(data_source="test").frame() .. raw:: html
estimator ridge_large_alpha ridge ridge_feature_engineering default_rf regularized_rf hgbr hgbr_early_stopping more_data
metric
r2 0.040223 0.609422 0.639733 0.784944 0.762550 0.831857 0.826642 0.833391
rmse 1.150918 0.734197 0.705134 0.544797 0.572461 0.481724 0.489138 0.479521
mae 0.907935 0.539913 0.508096 0.361575 0.394609 0.313363 0.316511 0.313229
mape 0.614893 0.320523 0.301925 0.206075 0.226745 0.175758 0.177941 0.176323
fit_time 0.002820 0.002704 0.002891 5.249770 1.802544 2.507134 0.713126 NaN
predict_time 0.001038 0.000795 0.001024 0.070329 0.027588 0.128328 0.036498 0.015607


.. GENERATED FROM PYTHON SOURCE LINES 320-332 Conclusion ========== SKD002 and SKD001 are two different consequences of a unsuited expressiveness. We relax regularization / add capacity and use informative features until the model beats a weak baseline; then we regularize, stop early, and add data if train scores have a noticeable gap compared to the test scores. Feature engineering helps the underfit side of the path; capacity control and more data help the overfit side. Hyperparameter choices play a dual role. In practice we combine several of these levers and re-run the checks until the results are satisfying enough. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 15.177 seconds) .. _sphx_glr_download_auto_examples_pitfalls_and_solutions_plot_skd001_skd002_overfitting_underfitting.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_skd001_skd002_overfitting_underfitting.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_skd001_skd002_overfitting_underfitting.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_skd001_skd002_overfitting_underfitting.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_