{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n\n# SKD014 & SKD015: Hyperparameter search pitfalls\n\nThis example walks through mitigations when checks\n`SKD014 ` and\n`SKD015 ` fire on a fitted search\nobject. SKD014 is an issue: numeric ``best_params_`` sit on the minimum or\nmaximum value tried, so the true optimum may lie outside the searched range.\nSKD015 is a tip: a hyperparameter is missing from the search space, which\nis incomplete rather than necessarily wrong.\n\nUsually, those tests are related to the search CV object and we advocate to\naddress them jointly when both fire.\n\nMitigations from the `automated_checks` user guide:\n\n**SKD014: hyperparameters at search edge** (issue)\n\n- extend ``param_grid`` or ``param_distributions`` beyond the flagged bounds,\n- for :class:`~sklearn.model_selection.RandomizedSearchCV`, increase ``n_iter``\n and sample from a wider range,\n- if SKD015 also fires, widen the search on every recommended hyperparameter.\n\n**SKD015: hyperparameters worth tuning** (tip)\n\n- add the suggested parameters to ``param_grid`` or ``param_distributions``.\n\nIn this example, we tune a\n:class:`~sklearn.ensemble.HistGradientBoostingClassifier` inside\n:func:`~skrub.tabular_pipeline` on a stratified subsample of the employee\nsalaries dataset (above-median salary as the positive class). The walkthrough\nhas three parts: missing hyperparameters (SKD015), edge hits (SKD014), then\none joint fix that clears both.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load the employee salaries dataset\n\nThe raw target is continuous salary. We turn the regression problem into a\nbinary classification task: predict whether an employee earns more than the\nmedian salary among employees in the dataset. Mixed HR features suit\n:func:`~skrub.tabular_pipeline`. A 3,000-row stratified subsample keeps the\ngallery grids short while preserving class balance.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\nfrom skrub.datasets import fetch_employee_salaries\n\ndataset = fetch_employee_salaries()\nX_full = dataset.X\ny_full = (dataset.y > dataset.y.median()).astype(int).rename(\"high_earner\")\n\nX, _, y, _ = train_test_split(\n X_full,\n y_full,\n train_size=3_000,\n stratify=y_full,\n random_state=42,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us inspect predictors and the binary target with\n:class:`~skrub.TableReport`.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from skrub import TableReport\n\nTableReport(X)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "TableReport(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Shared search setup\n\nLet us wrap HGB in :func:`~skrub.tabular_pipeline`. Early stopping lets a\nwide ``max_iter`` grid pick an interior budget in the first beat below. The\nouter hold-out uses :class:`~skore.TrainTestSplit` when we call\n:func:`~skore.evaluate`; each\n:class:`~sklearn.model_selection.GridSearchCV` below sets its own inner\n``cv``.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.ensemble import HistGradientBoostingClassifier\nfrom skrub import tabular_pipeline\n\nbase_pipeline = tabular_pipeline(\n HistGradientBoostingClassifier(\n max_iter=200,\n random_state=42,\n early_stopping=True,\n validation_fraction=0.1,\n n_iter_no_change=10,\n )\n)\nbase_pipeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Trigger SKD015: grid with only ``max_iter``\n\n``max_iter`` is a budget parameter, not a complexity knob in the SKD015 table.\nWe use :class:`~sklearn.model_selection.GridSearchCV` (not randomized search)\nso every candidate is evaluated and the run is reproducible. The grid is wide\non purpose: with early stopping, the best ``max_iter`` lands strictly inside\nthe list, so SKD014 stays quiet and this beat isolates the SKD015 tip about\nmissing recommended hyperparameters.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.model_selection import GridSearchCV\nfrom skore import TrainTestSplit, evaluate\n\nsplitter = TrainTestSplit(test_size=0.2, random_state=42, stratify=y)\n\nmax_iter_only_search = GridSearchCV(\n base_pipeline,\n param_grid={\n \"histgradientboostingclassifier__max_iter\": [\n 10,\n 25,\n 50,\n 100,\n 200,\n 500,\n 1000,\n ],\n },\n cv=3,\n scoring=\"neg_log_loss\",\n n_jobs=4,\n refit=True,\n)\n\nreport = evaluate(\n max_iter_only_search,\n X=X,\n y=y,\n pos_label=1,\n splitter=splitter,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the Tips tab, SKD015 should tip that learning rate, depth, and leaf size\nwere not searched. SKD014 should not fire here: the best ``max_iter`` is not\nthe minimum or maximum of the grid above. ``best_params_`` only contains\n``max_iter``: that incompleteness is the point. A search that only tweaks\ntraining budget ignores the hyperparameters that usually move generalization\nfor tree ensembles.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.checks.summarize(fast_mode=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.estimator_.best_params_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Trigger SKD014: two-point ``GridSearchCV``\n\nWith exactly two values on each searched hyperparameter, whichever value wins\nis always the tried minimum or maximum, so SKD014 fires deterministically. We\nstill omit depth / leaf hyperparameters so SKD015 tips as well. Prefer a small\ngrid over :class:`~sklearn.model_selection.RandomizedSearchCV` here: every\ncandidate is evaluated, and the edge story does not depend on which draws were\nsampled.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "edge_search = GridSearchCV(\n base_pipeline,\n param_grid={\n \"histgradientboostingclassifier__learning_rate\": [0.05, 0.1],\n \"histgradientboostingclassifier__max_iter\": [100, 200],\n },\n cv=3,\n scoring=\"neg_log_loss\",\n n_jobs=4,\n refit=True,\n)\n\nreport_edge = evaluate(\n edge_search,\n X=X,\n y=y,\n pos_label=1,\n splitter=splitter,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SKD014 should list numeric parameters at search edges as an issue; in the Tips\ntab, SKD015 should tip because depth / leaf hyperparameters are still missing.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_edge.checks.summarize(fast_mode=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_edge.estimator_.best_params_" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_edge.metrics.summarize(data_source=\"both\").frame()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# SKD014 & SKD015: widen bounds and add recommended hyperparameters\n\nLet us pad beyond the previous two-point edges so those learning-rate values\nbecome interior grid points, and add ``max_depth`` so SKD015 clears\n(``max_depth`` covers the tree-complexity family). ``None`` in ``max_depth``\nis non-numeric, so SKD014 ignores that hyperparameter and only watches\nlearning rate: fewer ways for the gallery to flake.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "full_search = GridSearchCV(\n base_pipeline,\n param_grid={\n \"histgradientboostingclassifier__learning_rate\": [0.01, 0.05, 0.1, 0.2],\n \"histgradientboostingclassifier__max_depth\": [3, 5, None],\n },\n cv=3,\n scoring=\"neg_log_loss\",\n n_jobs=4,\n refit=True,\n)\n\nreport_full = evaluate(\n full_search,\n X=X,\n y=y,\n pos_label=1,\n splitter=splitter,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SKD015 should clear once a recommended complexity hyperparameter is present\nwith learning rate. SKD014 clears when the best learning rate sits strictly\ninside ``[0.01, 0.05, 0.1, 0.2]`` (not at the padded ends). Check\n``best_params_`` against the grid above.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_full.checks.summarize(fast_mode=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_full.estimator_.best_params_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Compare search strategies\n\nHold-out log-loss and ROC AUC for the incomplete grid, the two-point edge\ngrid, and the joint fix. Clearing the findings means the *search design*\nimproved; still judge models on validation metrics and cost, not check status\nalone.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from skore import compare\n\nmetrics = (\n compare(\n {\n \"max_iter_only\": report,\n \"two_point_edge_grid\": report_edge,\n \"padded_recommended_params\": report_full,\n }\n )\n .metrics.summarize(data_source=\"both\", metric=[\"log_loss\", \"roc_auc\"])\n .frame()\n)\nmetrics.transpose()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n\nSKD015 tips about incomplete search spaces; SKD014 issues when optima stick to\nthe boundary of the box you tried. In this walkthrough, a ``max_iter``-only\nsearch missed key hyperparameters, a two-point grid forced edge hits, and one\npadded complete grid addressed both. Expand the search before deploying\n``best_params_``: passing checks are about search hygiene, not a guarantee of\nthe best model.\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.7" } }, "nbformat": 4, "nbformat_minor": 0 }