{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# SKD009 - Model performance vs. HistGradientBoosting baseline\n\nThis example walks through mitigations for check\n`SKD009 <skd009-worse-than-baseline>`. The check trains a strong\n:func:`~skrub.tabular_pipeline` baseline (HistGradientBoosting on vectorized\ndata). It always reports that baseline's performance, and warns when the\nestimator is significantly worse on default metrics.\n\nMitigations from the `user guide <skd009-worse-than-baseline>`, in the order\nwe try them here:\n\n- revisit feature engineering and preprocessing,\n- check whether the model family is appropriate,\n- switch to a stronger default such as HistGradientBoostingRegressor,\n- tune the model (here: moderated HGB capacity plus a log target).\n\nWe use the medical charge dataset with provider IDs and leakage columns removed.\nWe will see how applying these recommendations leads to a model that improves on the\nbaseline.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Load the medical charge dataset\n\n:func:`skrub.datasets.fetch_medical_charge` returns hospital billing records.\nWe drop provider identifiers and columns that leak the target, then subsample\n2,000 rows for a challenging regression task.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub.datasets import fetch_medical_charge\n\ndataset = fetch_medical_charge()\nX_full, y_full = dataset.X, dataset.y\n\nid_cols = [\n    \"Provider_Zip_Code\",\n    \"Provider_Id\",\n    \"Provider_Name\",\n    \"Provider_Street_Address\",\n]\nleakage_cols = [\"Average_Covered_Charges\", \"Average_Medicare_Payments\"]\n\nX = X_full.drop(columns=id_cols + leakage_cols).sample(2_000, random_state=42)\ny = y_full.loc[X.index]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Inspect the features matrix with :class:`~skrub.TableReport`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport\n\nTableReport(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The target is continuous total payment.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "TableReport(y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import TrainTestSplit\n\nsplitter = TrainTestSplit(random_state=42, test_size=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Trigger SKD009 with a linear pipeline\n\nStart with :func:`~skrub.tabular_pipeline` around\n:class:`~sklearn.linear_model.Ridge` so that encoding and imputation are\nalready in place before fitting the linear model.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import Ridge\nfrom skore import evaluate\nfrom skrub import tabular_pipeline\n\nreport_ridge = evaluate(tabular_pipeline(Ridge()), X=X, y=y, splitter=splitter)\nreport_ridge"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Find ``SKD009`` in the Tips tab below: the Ridge pipeline reports worse-than-baseline\nperformance on a majority of metrics.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_ridge.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Revisit feature engineering\n\nA linear model benefits from spelling out structure that trees would discover\nfrom the raw columns:\n\n- ``log1p(Total_Discharges)`` compresses a skewed volume signal (then drop the\n  raw count to avoid collinearity),\n- extract the numeric DRG code from labels like ``\"178 - ... W CC\"``,\n- flag severity markers in the text (``W MCC`` / ``W CC``), which affect\n  payment levels.\n\nKeep everything inside a pipeline so the same transforms run at predict time\non new data. Since this transformation is stateless, i.e., no information\nneeds to be computed and stored during training, we can safely use a\n:class:`~sklearn.preprocessing.FunctionTransformer`.\n\nWe also break down the :func:`~skrub.tabular_pipeline` into its components (vectorization of data,\nmissing value imputation and scaling) to keep the :class:`~sklearn.pipeline.Pipeline` flat.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import FunctionTransformer\nfrom skrub import SquashingScaler, TableVectorizer\n\n\ndef engineer_features(X):\n    \"\"\"Stateless feature engineering.\"\"\"\n    drg = X[\"DRG_Definition\"].str.upper()\n    return X.drop(columns=[\"Total_Discharges\"]).assign(\n        log_Total_Discharges=np.log1p(X[\"Total_Discharges\"]),\n        DRG_Code=(\n            X[\"DRG_Definition\"].str.extract(r\"^(\\d+)\", expand=False).astype(float)\n        ),\n        has_MCC=drg.str.contains(\"W MCC\", regex=False).astype(int),\n        has_CC=(\n            drg.str.contains(\"W CC\", regex=False)\n            & ~drg.str.contains(\"W MCC\", regex=False)\n        ).astype(int),\n    )\n\n\nridge_with_fe = make_pipeline(\n    FunctionTransformer(engineer_features),\n    TableVectorizer(),\n    SimpleImputer(add_indicator=True),\n    SquashingScaler(max_absolute_value=5),\n    Ridge(),\n)\n\nreport_ridge_fe = evaluate(ridge_with_fe, X=X, y=y, splitter=splitter)\nreport_ridge_fe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "With this feature engineering, we improved the test score.\n\nHowever, SKD009 still flags the model because it cannot beat the HGB baseline\non a majority of scores.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_ridge_fe.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Change model family: try a tree-based model\n\nIf nonlinearity and interactions matter, trees should close much of the gap.\nLet's compare a :class:`~sklearn.ensemble.RandomForestRegressor` pipeline to the\nengineered Ridge on the same split.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestRegressor\nfrom skore import compare\n\nreport_rf = evaluate(\n    tabular_pipeline(\n        RandomForestRegressor(\n            n_estimators=100,\n            min_samples_leaf=5,\n            random_state=42,\n            n_jobs=4,\n        )\n    ),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport_rf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "comparison_families = compare(\n    {\n        \"ridge_with_fe\": report_ridge_fe,\n        \"random_forest\": report_rf,\n    }\n)\ncomparison_families.metrics.summarize().frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this case, we see that we get closer to the gradient boosting baseline but we still\nhave a small gap (SKD009 reports an r2 of around 0.88 for the baseline).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_rf.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Switch to HistGradientBoostingRegressor\n\nskore's SKD009 performance baseline is itself an HGB pipeline. Matching that\nfamily is the natural next step once trees look promising, and we find that we are\non par with skore's baseline.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import HistGradientBoostingRegressor\n\nreport_hgb = evaluate(\n    tabular_pipeline(HistGradientBoostingRegressor(random_state=42)),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport_hgb"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_hgb.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Combine levers: features, HGB, and a log target\n\nLet's go a little further in our modelling efforts in order to beat that performant\nbaseline. For that, let us stack the previous techniques we used:\n\n- the engineered features,\n- an HGB with tuned parameters,\n- :class:`~sklearn.compose.TransformedTargetRegressor` with ``log1p`` /\n  ``expm1``, because payment totals are heavy-tailed.\n\nIn practice you would usually tune these knobs with\n:class:`~sklearn.model_selection.RandomizedSearchCV` or\n:class:`~sklearn.model_selection.GridSearchCV` from scikit-learn. To keep the\nexample short and reproducible, we pin one search outcome that beats the baseline\non this split.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.compose import TransformedTargetRegressor\nfrom skrub import ToCategorical\n\ntuned = TransformedTargetRegressor(\n    regressor=make_pipeline(\n        FunctionTransformer(engineer_features),\n        TableVectorizer(low_cardinality=ToCategorical()),\n        HistGradientBoostingRegressor(\n            learning_rate=0.05,\n            max_iter=500,\n            max_depth=5,\n            max_leaf_nodes=63,\n            min_samples_leaf=10,\n            l2_regularization=0.1,\n            random_state=42,\n        ),\n    ),\n    func=np.log1p,\n    inverse_func=np.expm1,\n)\n\nreport_tuned = evaluate(tuned, X=X, y=y, splitter=splitter)\nreport_tuned"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Our model is now significantly better than an HGB baseline, and SKD009 is still\nreporting baseline scores for reference.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tuned.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Conclusion\n\nSKD009 compares a model with a HistGradientBoosting baseline. It warns when\nthe model is significantly worse, and it still reports the baseline scores\nwhen the model is on par or better. The mitigations shown here are feature\nand target engineering, a change of model family, and hyperparameter tuning.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}