SKD001 & SKD002 - Overfitting and underfitting

SKD002 and SKD001 describe opposite ends of the same problem: model expressiveness.

• Too little expressiveness: train and test scores are close to those of a model that guesses randomly (often called a dummy model) (SKD002).

• Too much expressiveness: train scores pull far ahead of test scores (SKD001).

A performant model learns from the training data without memorizing its specificities, so that learned knowledge can be generalized to new data. This notebook walks the underfitting to overfitting path while showing different mitigations techniques:

• tweaking model capacity (model family / complexity)

• feature engineering

• tuning regularization

• using early stopping

• using more data

We use California housing (median house value). Half of the rows feed the main walkthrough; the other half is reserved to show the effect of adding training data.

Load the California housing dataset

Each row is a census block group in a geographical region whose coordinates we have (Latitude, Longitude). The features mix house characteristics (age, rooms, occupancy) with regional aggregates (median income, population). The target MedHouseVal is the median house value for the block group. The feature set is small but rich: later we combine columns such as AveRooms / AveOccup into a rooms-per-person marker.

import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

housing = fetch_california_housing(as_frame=True)

X, X_heldout, y, y_heldout = train_test_split(
    housing.data,
    housing.target,
    train_size=0.5,
    random_state=42,
)

Let us glance at the table. Pay attention to AveRooms and AveOccup: we will combine them into a rooms-per-person feature below.

from skrub import TableReport

TableReport(X)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

MedHouseVal is capped at 5.0 (500k USD). Click the column in the report to see the histogram and the pile-up at the ceiling.

TableReport(y)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We use the same split for every model comparison, with a fixed seed, so that we are sure that we are comparing models on the same test data.

from skore import TrainTestSplit

splitter = TrainTestSplit(test_size=0.2, random_state=42)

Underfitting: SKD002 fires

We start with a linear model that is overly regularized. Ridge with a very large alpha shrinks coefficients toward zero, so predictions stay close to a dummy baseline.

import skore
from sklearn.linear_model import Ridge

report_underfit = skore.evaluate(
    Ridge(alpha=1e6),
    X=X,
    y=y,
    splitter=splitter,
)
report_underfit.metrics.summarize(data_source="both").frame()

	Ridge (train)	Ridge (test)
metric
r2	0.041240	0.040223
rmse	1.127863	1.150918
mae	0.889223	0.907935
mape	0.607538	0.614893
fit_time	0.002285	0.002285
predict_time	0.000733	0.000739

For a model that learns so little, SKD002 fires.

report_underfit.checks.summarize(fast_mode=True)

Increase model expressiveness

Moving away from underfitting means giving the model enough expressiveness to use the inputs. Let us drop to a Ridge with a much smaller alpha: it already learns useful weights on each feature and is enough to clear SKD002 on this table.

report_ridge = skore.evaluate(Ridge(alpha=1.0), X=X, y=y, splitter=splitter)
report_ridge.metrics.summarize(data_source="both").frame()

	Ridge (train)	Ridge (test)
metric
r2	0.610830	0.609422
rmse	0.718573	0.734197
mae	0.525393	0.539913
mape	0.313230	0.320523
fit_time	0.003204	0.003204
predict_time	0.000873	0.000880

report_ridge.checks.summarize(fast_mode=True)

Feature engineering

Better features help when the model family is fine but the representation is weak (still an underfitting problem). Here, AveRooms / AveOccup is a rooms-per-person signal that a linear model can use more easily than the raw counts.

The same step has an overfitting side effect: every new feature also increases expressiveness. Mild engineering can close an underfit gap; aggressive expansions (very high-degree interactions, huge one-hot spaces) can reopen an overfit gap. Think of features as capacity you add to the inputs, not only to the estimator.

X_fe = X.assign(RoomsPerPerson=X["AveRooms"] / X["AveOccup"].clip(lower=0.1))

report_ridge_fe = skore.evaluate(Ridge(alpha=1.0), X=X_fe, y=y, splitter=splitter)
report_ridge_fe.metrics.summarize(data_source="both").frame()

	Ridge (train)	Ridge (test)
metric
r2	0.640718	0.639733
rmse	0.690429	0.705134
mae	0.494856	0.508096
mape	0.295383	0.301925
fit_time	0.002935	0.002935
predict_time	0.001219	0.001245

The test score rises while the train score stays close: we reduced underfitting without creating overfitting by adding a signal-rich feature.

report_ridge_fe.checks.summarize(fast_mode=True)

Overfitting: SKD001 fires

Hand-crafted features are one way to capture nonlinearity for a linear model. Another is to switch to a model family that learns nonlinear structure on its own. A default RandomForestRegressor does that, but unrestricted leaves can also memorize the training data: train metrics look excellent, test metrics improve but not as much as train ones, and SKD001 flags the gap.

from sklearn.ensemble import RandomForestRegressor

report_rf = skore.evaluate(
    RandomForestRegressor(random_state=42),
    X=X,
    y=y,
    splitter=splitter,
)
report_rf.metrics.summarize(data_source="both").frame()

	RandomForestRegressor (train)	RandomForestRegressor (test)
metric
r2	0.969057	0.784944
rmse	0.202621	0.544797
mae	0.134065	0.361575
mape	0.077009	0.206075
fit_time	4.870742	4.870742
predict_time	0.176020	0.066465

report_rf.checks.summarize(fast_mode=True)

Regularization

Once SKD001 appears, we pull expressiveness back with capacity limits on the estimator (leaf size, feature fraction, depth, learning rate, …).

Here, we set the hyperparameters of the model by hand. In practice, it is rather hard to know in advance which combination of hyperparameters leads to generalization. One should lean towards tuning hyperparameters. We would advocate for randomized search, with successive halving when the amount of samples is large. See tuning the hyper-parameters of an estimator. In this particular example, we are not implementing the search in order to keep the execution time of the example short.

model_rf_reg = RandomForestRegressor(
    min_samples_leaf=20,
    max_features=0.5,
    random_state=42,
)
report_rf_reg = skore.evaluate(
    model_rf_reg,
    X=X,
    y=y,
    splitter=splitter,
)
report_rf_reg.metrics.summarize(data_source="both").frame()

	RandomForestRegressor (train)	RandomForestRegressor (test)
metric
r2	0.809771	0.762550
rmse	0.502389	0.572461
mae	0.344942	0.394609
mape	0.200611	0.226745
fit_time	1.707768	1.707768
predict_time	0.080103	0.031034

report_rf_reg.checks.summarize(fast_mode=True)

Early stopping

For iterative learners, early stopping is another way to limit expressiveness that does not require a hyperparameter search. We hold out a validation set, monitor a metric, and stop when that metric stops improving. Further iterations are assumed to overfit. We can either pass a validation set to .fit or use the validation_fraction parameter. See scikit-learn’s example on gradient boosting with early stopping.

Let us compare a HistGradientBoostingRegressor with and without early stopping on the same split to see the effect. We bump max_iter so that early stopping has room to act. With validation_fraction, the early-stopped model trains on only 90% of each train fold; a small test drop relative to the unstopped run can come from that, rather than from stopping itself. Passing an external validation set (for example X_heldout / y_heldout) would avoid that, but wiring fit parameters through skore currently needs a skrub DataOp, so we keep validation_fraction here.

from sklearn.ensemble import HistGradientBoostingRegressor

report_hgbr = skore.evaluate(
    HistGradientBoostingRegressor(max_iter=1000, random_state=42),
    X=X,
    y=y,
    splitter=splitter,
)

model_hgbr_es = HistGradientBoostingRegressor(
    early_stopping=True,
    validation_fraction=0.1,
    n_iter_no_change=10,
    max_iter=1000,
    random_state=42,
)
report_hgbr_es = skore.evaluate(
    model_hgbr_es,
    X=X,
    y=y,
    splitter=splitter,
)

skore.compare(
    {
        "hgbr": report_hgbr,
        "hgbr_early_stopping": report_hgbr_es,
    }
).metrics.summarize(data_source="both").frame()

	hgbr (train)	hgbr (test)	hgbr_early_stopping (train)	hgbr_early_stopping (test)
metric
r2	0.990605	0.831857	0.935434	0.826642
rmse	0.111650	0.481724	0.292687	0.489138
mae	0.078763	0.313363	0.199150	0.316511
mape	0.046624	0.175758	0.114104	0.177941
fit_time	2.604526	2.604526	0.808667	0.808667
predict_time	0.364234	0.121463	0.110828	0.027179

report_hgbr.checks.summarize(fast_mode=True)

report_hgbr_es.checks.summarize(fast_mode=True)

More training data

Extra labeled rows help both regimes, but they are especially useful against overfitting: the model has fewer opportunities to memorize a small sample. Let us keep one fixed test fold, fit on the original train fold, then refit after concatenating the held-out half of the dataset. We use evaluate() with splitter="prefit" once the estimator is fitted. A side-effect is that fit time is unavailable (skore did not time .fit), so that metric appears as NaN in the reported tables.

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model_less = HistGradientBoostingRegressor(
    early_stopping=True,
    validation_fraction=0.1,
    n_iter_no_change=10,
    random_state=42,
).fit(X_train, y_train)
report_less = skore.evaluate(model_less, X_test, y_test, splitter="prefit")

model_more = HistGradientBoostingRegressor(
    early_stopping=True,
    validation_fraction=0.1,
    n_iter_no_change=10,
    random_state=42,
).fit(pd.concat([X_train, X_heldout]), pd.concat([y_train, y_heldout]))
report_more = skore.evaluate(model_more, X_test, y_test, splitter="prefit")

skore.compare(
    {"less_training_data": report_less, "more_training_data": report_more}
).metrics.summarize(data_source="test").frame()

estimator	less_training_data	more_training_data
metric
r2	0.820257	0.833391
rmse	0.498065	0.479521
mae	0.327594	0.313229
mape	0.185737	0.176323
predict_time	0.016623	0.013071

In production, a learning curve (many refits on growing subsets) shows whether more labels are still worth the cost. skore does not wrap that API yet; we can treat it as a scikit-learn-side diagnostic next to these checks.

Summary comparison

Here is a side-by-side comparison of the impact of the different techniques we applied, measured on the same test set.

skore.compare(
    {
        "ridge_large_alpha": report_underfit,
        "ridge": report_ridge,
        "ridge_feature_engineering": report_ridge_fe,
        "default_rf": report_rf,
        "regularized_rf": report_rf_reg,
        "hgbr": report_hgbr,
        "hgbr_early_stopping": report_hgbr_es,
        "more_data": report_more,
    }
).metrics.summarize(data_source="test").frame()

estimator	ridge_large_alpha	ridge	ridge_feature_engineering	default_rf	regularized_rf	hgbr	hgbr_early_stopping	more_data
metric
r2	0.040223	0.609422	0.639733	0.784944	0.762550	0.831857	0.826642	0.833391
rmse	1.150918	0.734197	0.705134	0.544797	0.572461	0.481724	0.489138	0.479521
mae	0.907935	0.539913	0.508096	0.361575	0.394609	0.313363	0.316511	0.313229
mape	0.614893	0.320523	0.301925	0.206075	0.226745	0.175758	0.177941	0.176323
fit_time	0.002285	0.003204	0.002935	4.870742	1.707768	2.604526	0.808667	NaN
predict_time	0.000739	0.000880	0.001245	0.066465	0.031034	0.121463	0.027179	0.013071

Conclusion

SKD002 and SKD001 are two different consequences of a unsuited expressiveness. We relax regularization / add capacity and use informative features until the model beats a weak baseline; then we regularize, stop early, and add data if train scores have a noticeable gap compared to the test scores.

Feature engineering helps the underfit side of the path; capacity control and more data help the overfit side. Hyperparameter choices play a dual role. In practice we combine several of these levers and re-run the checks until the results are satisfying enough.