{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# SKD016 - Estimator not tuned\n\nThis example walks through mitigations when check\n`SKD016 <skd016-estimator-not-tuned>` tips on a plain estimator left at\nscikit-learn defaults. The check compares initialization parameters against a\ncurated table of high-impact hyperparameters and suggests axes worth tuning.\n\nMitigations from the `automated_checks` user guide:\n\n- wrap the estimator in :class:`~sklearn.model_selection.GridSearchCV` or\n  :class:`~sklearn.model_selection.RandomizedSearchCV` over the suggested\n  parameters,\n- or set sensible non-default values manually.\n\nWe use the employee salaries dataset (above-median salary as the positive\nclass) with a default :func:`~skrub.tabular_pipeline` classifier. The goal is\nto move off factory defaults either through search or hand-picked values.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Load the employee salaries dataset\n\nMixed HR features suit ``tabular_pipeline``. A default\n``tabular_pipeline(\"classifier\")`` leaves\n:class:`~sklearn.ensemble.HistGradientBoostingClassifier` at sklearn\ndefaults \u2014 the setup SKD016 is designed to flag.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub.datasets import fetch_employee_salaries\n\ndataset = fetch_employee_salaries()\nX = dataset.X\ny_salary = dataset.y.squeeze()\ny = (y_salary > y_salary.median()).astype(int).rename(\"high_earner\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Inspect inputs and the binary target with :class:`~skrub.TableReport`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport\n\nTableReport(X)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "TableReport(y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We use the same stratified split for every comparison.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import TrainTestSplit\n\nsplitter = TrainTestSplit(test_size=0.2, random_state=42, stratify=y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Trigger SKD016 - untuned default pipeline\n\nDefaults are fine for a first look at the table, but they are not a production\nconfiguration. SKD016 names the high-impact axes that usually matter first for\nthis estimator family.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import skore\nfrom skrub import tabular_pipeline\n\nreport = skore.evaluate(\n    tabular_pipeline(\"classifier\"),\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter,\n)\nreport"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD016 should tip that HistGradientBoostingClassifier remains at defaults.\nRead which parameters it lists \u2014 the next sections search or set those knobs.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.checks.summarize()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Wrap the estimator in RandomizedSearchCV\n\nSearch the axes SKD016 typically flags for HGB (learning rate, iteration\nbudget, depth, leaf size) instead of accepting sklearn defaults. Once the\nreport wraps a fitted search object, SKD016 clears.\n\nA tuned search can still raise `SKD014 <skd014-hyperparams-at-search-edge>`\nor `SKD015 <skd015-hyperparameters-worth-tuning>` if the box is too narrow\nor incomplete \u2014 see that combined example.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from scipy.stats import loguniform, randint\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom sklearn.model_selection import RandomizedSearchCV\n\nbase_pipeline = tabular_pipeline(HistGradientBoostingClassifier(random_state=42))\n\nparam_distributions = {\n    \"histgradientboostingclassifier__learning_rate\": loguniform(1e-2, 2e-1),\n    \"histgradientboostingclassifier__max_iter\": randint(100, 401),\n    \"histgradientboostingclassifier__max_depth\": [3, 5, 8, None],\n    \"histgradientboostingclassifier__min_samples_leaf\": randint(10, 51),\n}\n\ntuned_search = RandomizedSearchCV(\n    base_pipeline,\n    param_distributions=param_distributions,\n    n_iter=8,\n    cv=3,\n    scoring=\"neg_log_loss\",\n    random_state=42,\n    refit=True,\n)\n\nreport_tuned = skore.evaluate(\n    tuned_search,\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter,\n)\nreport_tuned"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD016 should be absent; the report wraps a fitted search object.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tuned.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tuned.estimator_.best_params_"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Set sensible non-default values manually\n\nWhen a full search is impractical, hand-pick hyperparameters that differ from\ndefaults. SKD016 clears as soon as impactful knobs are no longer factory\nsettings \u2014 that is an intentional configuration signal, not proof that the\nvalues are optimal. Prefer validated search when you can afford it.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model_manual = tabular_pipeline(\n    HistGradientBoostingClassifier(\n        learning_rate=0.05,\n        max_iter=200,\n        max_depth=5,\n        min_samples_leaf=20,\n        random_state=42,\n    )\n)\n\nreport_manual = skore.evaluate(\n    model_manual,\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter,\n)\nreport_manual"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD016 should be absent once hyperparameters differ from defaults.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_manual.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_manual.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Compare mitigations\n\nHold-out metrics for the default pipeline and the hand-tuned one. The\nRandomizedSearchCV report wraps a search estimator, so its metric table can\nlook different in :func:`~skore.compare`; we show it in its own cell below.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "comparison = skore.compare(\n    {\n        \"default_pipeline\": report,\n        \"hand_tuned_hgb\": report_manual,\n    }\n)\ncomparison.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tuned.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Conclusion\n\nSKD016 nudges you off scikit-learn defaults for high-impact estimators.\nRandomized search and hand-tuned HGB parameters both clear the tip; clearing\nthe check means you left factory settings, not that the model is finished.\nPair manual choices with periodic search, and watch SKD014/SKD015 once you\nwrap a ``BaseSearchCV``.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}