{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# SKD001 & SKD002 - Overfitting and underfitting\n\n`SKD002 <skd002-underfitting>` and `SKD001 <skd001-overfitting>`\ndescribe opposite ends of the same problem: model *expressiveness*.\n\n- Too little expressiveness: train and test scores are close to those of a\n  model that guesses randomly (often called a dummy model) (SKD002).\n- Too much expressiveness: train scores pull far ahead of test scores\n  (SKD001).\n\nA performant model learns from the training data without memorizing its\nspecificities, so that learned knowledge can be generalized to new data. This\nnotebook walks the underfitting to overfitting path while showing different\nmitigations techniques:\n\n- tweaking model capacity (model family / complexity)\n- feature engineering\n- tuning regularization\n- using early stopping\n- using more data\n\nWe use California housing (median house value). Half of the rows feed the main\nwalkthrough; the other half is reserved to show the effect of adding training\ndata.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Load the California housing dataset\n\nEach row is a census block group in a geographical region whose coordinates\nwe have (``Latitude``, ``Longitude``). The features mix house characteristics\n(age, rooms, occupancy) with regional aggregates (median income, population).\nThe target ``MedHouseVal`` is the median house value for the block group.\nThe feature set is small but rich: later we combine columns such as\n``AveRooms / AveOccup`` into a rooms-per-person marker.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import pandas as pd\nfrom sklearn.datasets import fetch_california_housing\nfrom sklearn.model_selection import train_test_split\n\nhousing = fetch_california_housing(as_frame=True)\n\nX, X_heldout, y, y_heldout = train_test_split(\n    housing.data,\n    housing.target,\n    train_size=0.5,\n    random_state=42,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let us glance at the table. Pay attention to ``AveRooms`` and ``AveOccup``:\nwe will combine them into a rooms-per-person feature below.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport\n\nTableReport(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "``MedHouseVal`` is capped at 5.0 (500k USD). Click the column in the report to\nsee the histogram and the pile-up at the ceiling.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "TableReport(y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We use the same split for every model comparison, with a fixed seed, so that\nwe are sure that we are comparing models on the same test data.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import TrainTestSplit\n\nsplitter = TrainTestSplit(test_size=0.2, random_state=42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Underfitting: SKD002 fires\n\nWe start with a linear model that is overly regularized.\n:class:`~sklearn.linear_model.Ridge` with a very large ``alpha`` shrinks\ncoefficients toward zero, so predictions stay close to a dummy baseline.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import skore\nfrom sklearn.linear_model import Ridge\n\nreport_underfit = skore.evaluate(\n    Ridge(alpha=1e6),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport_underfit.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For a model that learns so little, SKD002 fires.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_underfit.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Increase model expressiveness\n\nMoving away from underfitting means giving the model enough expressiveness to\nuse the inputs. Let us drop to a :class:`~sklearn.linear_model.Ridge` with a\nmuch smaller ``alpha``: it already learns useful weights on each feature and\nis enough to clear SKD002 on this table.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_ridge = skore.evaluate(Ridge(alpha=1.0), X=X, y=y, splitter=splitter)\nreport_ridge.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_ridge.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Feature engineering\n\nBetter features help when the model family is fine but the representation\nis weak (still an underfitting problem). Here, ``AveRooms / AveOccup`` is a\nrooms-per-person signal that a linear model can use more easily than the raw\ncounts.\n\nThe same step has an overfitting side effect: every new feature also increases\nexpressiveness. Mild engineering can close an underfit gap; aggressive\nexpansions (very high-degree interactions, huge one-hot spaces) can reopen an\noverfit gap. Think of features as capacity you add to the inputs, not only\nto the estimator.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "X_fe = X.assign(RoomsPerPerson=X[\"AveRooms\"] / X[\"AveOccup\"].clip(lower=0.1))\n\nreport_ridge_fe = skore.evaluate(Ridge(alpha=1.0), X=X_fe, y=y, splitter=splitter)\nreport_ridge_fe.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The test score rises while the train score stays close: we reduced underfitting\nwithout creating overfitting by adding a signal-rich feature.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_ridge_fe.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Overfitting: SKD001 fires\n\nHand-crafted features are one way to capture nonlinearity for a linear model.\nAnother is to switch to a model family that learns nonlinear structure on its\nown. A default :class:`~sklearn.ensemble.RandomForestRegressor` does that,\nbut unrestricted leaves can also memorize the training data: train metrics\nlook excellent, test metrics improve but not as much as train ones, and\nSKD001 flags the gap.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestRegressor\n\nreport_rf = skore.evaluate(\n    RandomForestRegressor(random_state=42),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport_rf.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_rf.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Regularization\n\nOnce SKD001 appears, we pull expressiveness back with capacity limits on the\nestimator (leaf size, feature fraction, depth, learning rate, \u2026).\n\nHere, we set the hyperparameters of the model by hand. In practice, it is\nrather hard to know in advance which combination of hyperparameters leads to\ngeneralization. One should lean towards tuning hyperparameters. We would\nadvocate for randomized search, with successive halving when the amount of\nsamples is large. See [tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html#tuning-the-hyper-parameters-of-an-estimator).\nIn this particular example, we are not implementing the search in order to\nkeep the execution time of the example short.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model_rf_reg = RandomForestRegressor(\n    min_samples_leaf=20,\n    max_features=0.5,\n    random_state=42,\n)\nreport_rf_reg = skore.evaluate(\n    model_rf_reg,\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport_rf_reg.metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_rf_reg.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Early stopping\n\nFor iterative learners, early stopping is another way to limit expressiveness\nthat does not require a hyperparameter search. We hold out a validation set,\nmonitor a metric, and stop when that metric stops improving. Further\niterations are assumed to overfit. We can either pass a validation set to\n``.fit`` or use the ``validation_fraction`` parameter. See scikit-learn's\nexample on [gradient boosting with early stopping](https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_early_stopping.html).\n\nLet us compare a :class:`~sklearn.ensemble.HistGradientBoostingRegressor`\nwith and without early stopping on the same split to see the effect. We bump\n``max_iter`` so that early stopping has room to act. With\n``validation_fraction``, the early-stopped model trains on only 90% of each\ntrain fold; a small test drop relative to the unstopped run can come from\nthat, rather than from stopping itself. Passing an external validation set\n(for example ``X_heldout`` / ``y_heldout``) would avoid that, but wiring fit\nparameters through skore currently needs a skrub DataOp, so we keep\n``validation_fraction`` here.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import HistGradientBoostingRegressor\n\nreport_hgbr = skore.evaluate(\n    HistGradientBoostingRegressor(max_iter=1000, random_state=42),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\n\nmodel_hgbr_es = HistGradientBoostingRegressor(\n    early_stopping=True,\n    validation_fraction=0.1,\n    n_iter_no_change=10,\n    max_iter=1000,\n    random_state=42,\n)\nreport_hgbr_es = skore.evaluate(\n    model_hgbr_es,\n    X=X,\n    y=y,\n    splitter=splitter,\n)\n\nskore.compare(\n    {\n        \"hgbr\": report_hgbr,\n        \"hgbr_early_stopping\": report_hgbr_es,\n    }\n).metrics.summarize(data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_hgbr.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_hgbr_es.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# More training data\n\nExtra labeled rows help both regimes, but they are especially useful against\noverfitting: the model has fewer opportunities to memorize a small sample.\nLet us keep one fixed test fold, fit on the original train fold, then refit\nafter concatenating the held-out half of the dataset. We use\n:func:`~skore.evaluate` with ``splitter=\"prefit\"`` once the estimator is\nfitted. A side-effect is that fit time is unavailable (skore did not time\n``.fit``), so that metric appears as NaN in the reported tables.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)\n\nmodel_less = HistGradientBoostingRegressor(\n    early_stopping=True,\n    validation_fraction=0.1,\n    n_iter_no_change=10,\n    random_state=42,\n).fit(X_train, y_train)\nreport_less = skore.evaluate(model_less, X_test, y_test, splitter=\"prefit\")\n\nmodel_more = HistGradientBoostingRegressor(\n    early_stopping=True,\n    validation_fraction=0.1,\n    n_iter_no_change=10,\n    random_state=42,\n).fit(pd.concat([X_train, X_heldout]), pd.concat([y_train, y_heldout]))\nreport_more = skore.evaluate(model_more, X_test, y_test, splitter=\"prefit\")\n\nskore.compare(\n    {\"less_training_data\": report_less, \"more_training_data\": report_more}\n).metrics.summarize(data_source=\"test\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In production, a [learning curve](https://scikit-learn.org/stable/modules/learning_curve.html#learning-curve)\n(many refits on growing subsets) shows whether more labels are still worth\nthe cost. skore does not wrap that API yet; we can treat it as a\nscikit-learn-side diagnostic next to these checks.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Summary comparison\n\nHere is a side-by-side comparison of the impact of the different techniques we\napplied, measured on the same test set.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "skore.compare(\n    {\n        \"ridge_large_alpha\": report_underfit,\n        \"ridge\": report_ridge,\n        \"ridge_feature_engineering\": report_ridge_fe,\n        \"default_rf\": report_rf,\n        \"regularized_rf\": report_rf_reg,\n        \"hgbr\": report_hgbr,\n        \"hgbr_early_stopping\": report_hgbr_es,\n        \"more_data\": report_more,\n    }\n).metrics.summarize(data_source=\"test\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Conclusion\n\nSKD002 and SKD001 are two different consequences of a unsuited expressiveness.\nWe relax regularization / add capacity and use informative features until the\nmodel beats a weak baseline; then we regularize, stop early, and add data if\ntrain scores have a noticeable gap compared to the test scores.\n\nFeature engineering helps the underfit side of the path; capacity control\nand more data help the overfit side. Hyperparameter choices play a dual\nrole. In practice we combine several of these levers and re-run the checks\nuntil the results are satisfying enough.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}