SKD014 & SKD015: Hyperparameter search pitfalls

This example walks through mitigations when checks SKD014 and SKD015 fire on a fitted search object. SKD014 is an issue: numeric best_params_ sit on the minimum or maximum value tried, so the true optimum may lie outside the searched range. SKD015 is a tip: a hyperparameter is missing from the search space, which is incomplete rather than necessarily wrong.

Usually, those tests are related to the search CV object and we advocate to address them jointly when both fire.

Mitigations from the Automated checks user guide:

SKD014: hyperparameters at search edge (issue)

• extend param_grid or param_distributions beyond the flagged bounds,

• for RandomizedSearchCV, increase n_iter and sample from a wider range,

• if SKD015 also fires, widen the search on every recommended hyperparameter.

SKD015: hyperparameters worth tuning (tip)

• add the suggested parameters to param_grid or param_distributions.

In this example, we tune a HistGradientBoostingClassifier inside tabular_pipeline() on a stratified subsample of the employee salaries dataset (above-median salary as the positive class). The walkthrough has three parts: missing hyperparameters (SKD015), edge hits (SKD014), then one joint fix that clears both.

Load the employee salaries dataset

The raw target is continuous salary. We turn the regression problem into a binary classification task: predict whether an employee earns more than the median salary among employees in the dataset. Mixed HR features suit tabular_pipeline(). A 3,000-row stratified subsample keeps the gallery grids short while preserving class balance.

from sklearn.model_selection import train_test_split
from skrub.datasets import fetch_employee_salaries

dataset = fetch_employee_salaries()
X_full = dataset.X
y_full = (dataset.y > dataset.y.median()).astype(int).rename("high_earner")

X, _, y, _ = train_test_split(
    X_full,
    y_full,
    train_size=3_000,
    stratify=y_full,
    random_state=42,
)

Let us inspect predictors and the binary target with TableReport.

from skrub import TableReport

TableReport(X)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

TableReport(y)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Shared search setup

Let us wrap HGB in tabular_pipeline(). Early stopping lets a wide max_iter grid pick an interior budget in the first beat below. The outer hold-out uses TrainTestSplit when we call evaluate(); each GridSearchCV below sets its own inner cv.

from sklearn.ensemble import HistGradientBoostingClassifier
from skrub import tabular_pipeline

base_pipeline = tabular_pipeline(
    HistGradientBoostingClassifier(
        max_iter=200,
        random_state=42,
        early_stopping=True,
        validation_fraction=0.1,
        n_iter_no_change=10,
    )
)
base_pipeline

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=ToCategorical())),
                ('histgradientboostingclassifier',
                 HistGradientBoostingClassifier(early_stopping=True,
                                                max_iter=200,
                                                random_state=42))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Trigger SKD015: grid with only max_iter

max_iter is a budget parameter, not a complexity knob in the SKD015 table. We use GridSearchCV (not randomized search) so every candidate is evaluated and the run is reproducible. The grid is wide on purpose: with early stopping, the best max_iter lands strictly inside the list, so SKD014 stays quiet and this beat isolates the SKD015 tip about missing recommended hyperparameters.

from sklearn.model_selection import GridSearchCV
from skore import TrainTestSplit, evaluate

splitter = TrainTestSplit(test_size=0.2, random_state=42, stratify=y)

max_iter_only_search = GridSearchCV(
    base_pipeline,
    param_grid={
        "histgradientboostingclassifier__max_iter": [
            10,
            25,
            50,
            100,
            200,
            500,
            1000,
        ],
    },
    cv=3,
    scoring="neg_log_loss",
    n_jobs=4,
    refit=True,
)

report = evaluate(
    max_iter_only_search,
    X=X,
    y=y,
    pos_label=1,
    splitter=splitter,
)

In the Tips tab, SKD015 should tip that learning rate, depth, and leaf size were not searched. SKD014 should not fire here: the best max_iter is not the minimum or maximum of the grid above. best_params_ only contains max_iter: that incompleteness is the point. A search that only tweaks training budget ignores the hyperparameters that usually move generalization for tree ensembles.

report.checks.summarize(fast_mode=True)

report.estimator_.best_params_

{'histgradientboostingclassifier__max_iter': 50}

Trigger SKD014: two-point GridSearchCV

With exactly two values on each searched hyperparameter, whichever value wins is always the tried minimum or maximum, so SKD014 fires deterministically. We still omit depth / leaf hyperparameters so SKD015 tips as well. Prefer a small grid over RandomizedSearchCV here: every candidate is evaluated, and the edge story does not depend on which draws were sampled.

edge_search = GridSearchCV(
    base_pipeline,
    param_grid={
        "histgradientboostingclassifier__learning_rate": [0.05, 0.1],
        "histgradientboostingclassifier__max_iter": [100, 200],
    },
    cv=3,
    scoring="neg_log_loss",
    n_jobs=4,
    refit=True,
)

report_edge = evaluate(
    edge_search,
    X=X,
    y=y,
    pos_label=1,
    splitter=splitter,
)

SKD014 should list numeric parameters at search edges as an issue; in the Tips tab, SKD015 should tip because depth / leaf hyperparameters are still missing.

report_edge.checks.summarize(fast_mode=True)

report_edge.estimator_.best_params_

{'histgradientboostingclassifier__learning_rate': 0.05, 'histgradientboostingclassifier__max_iter': 100}

report_edge.metrics.summarize(data_source="both").frame()

	GridSearchCV (train)	GridSearchCV (test)
metric
score	-0.068762	-0.189901
accuracy	0.986667	0.933333
precision	0.985050	0.942177
recall	0.988333	0.923333
roc_auc	0.997754	0.977394
log_loss	0.068762	0.189901
brier_score	0.014418	0.052530
fit_time	10.189327	10.189327
predict_time	0.182861	0.083963

SKD014 & SKD015: widen bounds and add recommended hyperparameters

Let us pad beyond the previous two-point edges so those learning-rate values become interior grid points, and add max_depth so SKD015 clears (max_depth covers the tree-complexity family). None in max_depth is non-numeric, so SKD014 ignores that hyperparameter and only watches learning rate: fewer ways for the gallery to flake.

full_search = GridSearchCV(
    base_pipeline,
    param_grid={
        "histgradientboostingclassifier__learning_rate": [0.01, 0.05, 0.1, 0.2],
        "histgradientboostingclassifier__max_depth": [3, 5, None],
    },
    cv=3,
    scoring="neg_log_loss",
    n_jobs=4,
    refit=True,
)

report_full = evaluate(
    full_search,
    X=X,
    y=y,
    pos_label=1,
    splitter=splitter,
)

SKD015 should clear once a recommended complexity hyperparameter is present with learning rate. SKD014 clears when the best learning rate sits strictly inside [0.01, 0.05, 0.1, 0.2] (not at the padded ends). Check best_params_ against the grid above.

report_full.checks.summarize(fast_mode=True)

report_full.estimator_.best_params_

{'histgradientboostingclassifier__learning_rate': 0.05, 'histgradientboostingclassifier__max_depth': 5}

Compare search strategies

Hold-out log-loss and ROC AUC for the incomplete grid, the two-point edge grid, and the joint fix. Clearing the findings means the search design improved; still judge models on validation metrics and cost, not check status alone.

from skore import compare

metrics = (
    compare(
        {
            "max_iter_only": report,
            "two_point_edge_grid": report_edge,
            "padded_recommended_params": report_full,
        }
    )
    .metrics.summarize(data_source="both", metric=["log_loss", "roc_auc"])
    .frame()
)
metrics.transpose()

metric	log_loss	roc_auc
max_iter_only (train)	0.067503	0.997784
max_iter_only (test)	0.183736	0.978528
two_point_edge_grid (train)	0.068762	0.997754
two_point_edge_grid (test)	0.189901	0.977394
padded_recommended_params (train)	0.077473	0.997272
padded_recommended_params (test)	0.192798	0.976550

Conclusion

SKD015 tips about incomplete search spaces; SKD014 issues when optima stick to the boundary of the box you tried. In this walkthrough, a max_iter-only search missed key hyperparameters, a two-point grid forced edge hits, and one padded complete grid addressed both. Expand the search before deploying best_params_: passing checks are about search hygiene, not a guarantee of the best model.