{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# SKD009 - Model worse than baseline\n\nThis example walks through mitigations when check\n`SKD009 <skd009-worse-than-baseline>` fires. The check trains a strong\n:func:`~skrub.tabular_pipeline` baseline (gradient boosting on vectorized\ndata) and flags estimators that are significantly worse on default metrics.\n\nMitigations from the `user guide <skd009-worse-than-baseline>`, in the order\nwe try them here:\n\n- revisit feature engineering and preprocessing,\n- check whether the model family is appropriate,\n- switch to a stronger default such as HistGradientBoostingRegressor,\n- tune the model (here: moderated HGB capacity plus a log target).\n\nWe use the medical charge dataset with provider IDs and leakage columns removed.\nWe will see how applying these recommendations leads to a model that improves on the\nbaseline.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Load the medical charge dataset\n\n:func:`skrub.datasets.fetch_medical_charge` returns hospital billing records.\nWe drop provider identifiers and columns that leak the target, then subsample\n2,000 rows for a challenging regression task.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub.datasets import fetch_medical_charge\n\ndataset = fetch_medical_charge()\nX_full, y_full = dataset.X, dataset.y\n\nid_cols = [\n    \"Provider_Zip_Code\",\n    \"Provider_Id\",\n    \"Provider_Name\",\n    \"Provider_Street_Address\",\n]\nleakage_cols = [\"Average_Covered_Charges\", \"Average_Medicare_Payments\"]\n\nX = X_full.drop(columns=id_cols + leakage_cols).sample(2_000, random_state=42)\ny = y_full.loc[X.index]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Inspect the features matrix with :class:`~skrub.TableReport`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport\n\nTableReport(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The target is continuous total payment.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "TableReport(y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import TrainTestSplit\n\nsplitter = TrainTestSplit(random_state=42, test_size=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Trigger SKD009 with a linear pipeline\n\nStart with :func:`~skrub.tabular_pipeline` around\n:class:`~sklearn.linear_model.Ridge` so that encoding and imputation are\nalready in place before fitting the linear model.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import Ridge\nfrom skore import evaluate\nfrom skrub import tabular_pipeline\n\nreport_ridge = evaluate(tabular_pipeline(Ridge()), X=X, y=y, splitter=splitter)\nreport_ridge"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Find ``SKD009`` in the Tips tab below: the Ridge pipeline reports worse-than-baseline\nperformance on a majority of metrics.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_ridge.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Revisit feature engineering\n\nA linear model benefits from spelling out structure that trees would discover\nfrom the raw columns:\n\n- ``log1p(Total_Discharges)`` compresses a skewed volume signal (then drop the\n  raw count to avoid collinearity),\n- extract the numeric DRG code from labels like ``\"178 - ... W CC\"``,\n- flag severity markers in the text (``W MCC`` / ``W CC``), which affect\n  payment levels.\n\nKeep everything inside a pipeline so the same transforms run at predict time\non new data. Since this transformation is stateless, i.e., no information\nneeds to be computed and stored during training, we can safely use a\n:class:`~sklearn.preprocessing.FunctionTransformer`.\n\nWe also break down the :func:`~skrub.tabular_pipeline` into its components (vectorization of data,\nmissing value imputation and scaling) to keep the :class:`~sklearn.pipeline.Pipeline` flat.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import FunctionTransformer\nfrom skrub import SquashingScaler, TableVectorizer\n\n\ndef engineer_features(X):\n    \"\"\"Stateless feature engineering.\"\"\"\n    drg = X[\"DRG_Definition\"].str.upper()\n    return X.drop(columns=[\"Total_Discharges\"]).assign(\n        log_Total_Discharges=np.log1p(X[\"Total_Discharges\"]),\n        DRG_Code=(\n            X[\"DRG_Definition\"].str.extract(r\"^(\\d+)\", expand=False).astype(float)\n        ),\n        has_MCC=drg.str.contains(\"W MCC\", regex=False).astype(int),\n        has_CC=(\n            drg.str.contains(\"W CC\", regex=False)\n            & ~drg.str.contains(\"W MCC\", regex=False)\n        ).astype(int),\n    )\n\n\nridge_with_fe = make_pipeline(\n    FunctionTransformer(engineer_features),\n    TableVectorizer(),\n    SimpleImputer(add_indicator=True),\n    SquashingScaler(max_absolute_value=5),\n    Ridge(),\n)\n\nreport_ridge_fe = evaluate(ridge_with_fe, X=X, y=y, splitter=splitter)\nreport_ridge_fe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "With this feature engineering, we improved the test score.\n\nHowever, SKD009 still flags the model because it cannot beat the HGB baseline\non a majority of scores.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_ridge_fe.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Change model family: try a tree-based model\n\nIf nonlinearity and interactions matter, trees should close much of the gap.\nLet's compare a :class:`~sklearn.ensemble.RandomForestRegressor` pipeline to the\nengineered Ridge on the same split.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestRegressor\nfrom skore import compare\n\nreport_rf = evaluate(\n    tabular_pipeline(\n        RandomForestRegressor(\n            n_estimators=100,\n            min_samples_leaf=5,\n            random_state=42,\n            n_jobs=4,\n        )\n    ),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport_rf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "comparison_families = compare(\n    {\n        \"ridge_with_fe\": report_ridge_fe,\n        \"random_forest\": report_rf,\n    }\n)\ncomparison_families.metrics.summarize().frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this case, we see that we get closer to the gradient boosting baseline but we still\nhave a small gap (SKD009 reports an r2 of around 0.88 for the baseline).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_rf.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Switch to HistGradientBoostingRegressor\n\nskore's SKD009 performance baseline is itself an HGB pipeline. Matching that\nfamily is the natural next step once trees look promising, and we find that we are\non par with skore's baseline.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import HistGradientBoostingRegressor\n\nreport_hgb = evaluate(\n    tabular_pipeline(HistGradientBoostingRegressor(random_state=42)),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport_hgb"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_hgb.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Combine levers: features, HGB, and a log target\n\nLet's go a little further in our modelling efforts in order to beat that performant\nbaseline. For that, let us stack the previous techniques we used:\n\n- the engineered features,\n- an HGB with tuned parameters,\n- :class:`~sklearn.compose.TransformedTargetRegressor` with ``log1p`` /\n  ``expm1``, because payment totals are heavy-tailed.\n\nIn practice you would usually tune these knobs with\n:class:`~sklearn.model_selection.RandomizedSearchCV` or\n:class:`~sklearn.model_selection.GridSearchCV` from scikit-learn. To keep the\nexample short and reproducible, we pin one search outcome that beats the baseline\non this split.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.compose import TransformedTargetRegressor\nfrom skrub import ToCategorical\n\ntuned = TransformedTargetRegressor(\n    regressor=make_pipeline(\n        FunctionTransformer(engineer_features),\n        TableVectorizer(low_cardinality=ToCategorical()),\n        HistGradientBoostingRegressor(\n            learning_rate=0.05,\n            max_iter=500,\n            max_depth=5,\n            max_leaf_nodes=63,\n            min_samples_leaf=10,\n            l2_regularization=0.1,\n            random_state=42,\n        ),\n    ),\n    func=np.log1p,\n    inverse_func=np.expm1,\n)\n\nreport_tuned = evaluate(tuned, X=X, y=y, splitter=splitter)\nreport_tuned"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Our model is now significantly better than an HGB baseline, and SKD009 is still\nreporting baseline scores for reference.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tuned.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Conclusion\n\nSKD009 informs you when your model is beaten by a simple but strong baseline.\nWe showed different approaches to address it: feature and target engineering,\nchanging model family and tuning hyperparameters to finally clear well past\nthe baseline.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}