.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/pitfalls_and_solutions/plot_skd009_worse_than_baseline.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_pitfalls_and_solutions_plot_skd009_worse_than_baseline.py: .. _example_skd009_worse_than_baseline: SKD009 - Model performance vs. HistGradientBoosting baseline ============================================================ This example walks through mitigations for check :ref:`SKD009 `. The check trains a strong :func:`~skrub.tabular_pipeline` baseline (HistGradientBoosting on vectorized data). It always reports that baseline's performance, and warns when the estimator is significantly worse on default metrics. Mitigations from the :ref:`user guide `, in the order we try them here: - revisit feature engineering and preprocessing, - check whether the model family is appropriate, - switch to a stronger default such as HistGradientBoostingRegressor, - tune the model (here: moderated HGB capacity plus a log target). We use the medical charge dataset with provider IDs and leakage columns removed. We will see how applying these recommendations leads to a model that improves on the baseline. .. GENERATED FROM PYTHON SOURCE LINES 27-33 Load the medical charge dataset =============================== :func:`skrub.datasets.fetch_medical_charge` returns hospital billing records. We drop provider identifiers and columns that leak the target, then subsample 2,000 rows for a challenging regression task. .. GENERATED FROM PYTHON SOURCE LINES 33-50 .. code-block:: Python from skrub.datasets import fetch_medical_charge dataset = fetch_medical_charge() X_full, y_full = dataset.X, dataset.y id_cols = [ "Provider_Zip_Code", "Provider_Id", "Provider_Name", "Provider_Street_Address", ] leakage_cols = ["Average_Covered_Charges", "Average_Medicare_Payments"] X = X_full.drop(columns=id_cols + leakage_cols).sample(2_000, random_state=42) y = y_full.loc[X.index] .. rst-class:: sphx-glr-script-out .. code-block:: none Downloading 'medical_charge' from https://github.com/skrub-data/skrub-data-files/raw/refs/heads/main/medical_charge.zip (attempt 1/3) .. GENERATED FROM PYTHON SOURCE LINES 51-52 Inspect the features matrix with :class:`~skrub.TableReport`. .. GENERATED FROM PYTHON SOURCE LINES 52-57 .. code-block:: Python from skrub import TableReport TableReport(X) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 58-59 The target is continuous total payment. .. GENERATED FROM PYTHON SOURCE LINES 59-62 .. code-block:: Python TableReport(y) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 63-67 .. code-block:: Python from skore import TrainTestSplit splitter = TrainTestSplit(random_state=42, test_size=0.2) .. GENERATED FROM PYTHON SOURCE LINES 68-74 Trigger SKD009 with a linear pipeline ===================================== Start with :func:`~skrub.tabular_pipeline` around :class:`~sklearn.linear_model.Ridge` so that encoding and imputation are already in place before fitting the linear model. .. GENERATED FROM PYTHON SOURCE LINES 74-82 .. code-block:: Python from sklearn.linear_model import Ridge from skore import evaluate from skrub import tabular_pipeline report_ridge = evaluate(tabular_pipeline(Ridge()), X=X, y=y, splitter=splitter) report_ridge .. raw:: html
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                    ('simpleimputer', SimpleImputer(add_indicator=True)),
                    ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                    ('ridge', Ridge())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 83-85 Find ``SKD009`` in the Tips tab below: the Ridge pipeline reports worse-than-baseline performance on a majority of metrics. .. GENERATED FROM PYTHON SOURCE LINES 85-88 .. code-block:: Python report_ridge.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 89-108 Revisit feature engineering =========================== A linear model benefits from spelling out structure that trees would discover from the raw columns: - ``log1p(Total_Discharges)`` compresses a skewed volume signal (then drop the raw count to avoid collinearity), - extract the numeric DRG code from labels like ``"178 - ... W CC"``, - flag severity markers in the text (``W MCC`` / ``W CC``), which affect payment levels. Keep everything inside a pipeline so the same transforms run at predict time on new data. Since this transformation is stateless, i.e., no information needs to be computed and stored during training, we can safely use a :class:`~sklearn.preprocessing.FunctionTransformer`. We also break down the :func:`~skrub.tabular_pipeline` into its components (vectorization of data, missing value imputation and scaling) to keep the :class:`~sklearn.pipeline.Pipeline` flat. .. GENERATED FROM PYTHON SOURCE LINES 108-143 .. code-block:: Python import numpy as np from sklearn.impute import SimpleImputer from sklearn.pipeline import make_pipeline from sklearn.preprocessing import FunctionTransformer from skrub import SquashingScaler, TableVectorizer def engineer_features(X): """Stateless feature engineering.""" drg = X["DRG_Definition"].str.upper() return X.drop(columns=["Total_Discharges"]).assign( log_Total_Discharges=np.log1p(X["Total_Discharges"]), DRG_Code=( X["DRG_Definition"].str.extract(r"^(\d+)", expand=False).astype(float) ), has_MCC=drg.str.contains("W MCC", regex=False).astype(int), has_CC=( drg.str.contains("W CC", regex=False) & ~drg.str.contains("W MCC", regex=False) ).astype(int), ) ridge_with_fe = make_pipeline( FunctionTransformer(engineer_features), TableVectorizer(), SimpleImputer(add_indicator=True), SquashingScaler(max_absolute_value=5), Ridge(), ) report_ridge_fe = evaluate(ridge_with_fe, X=X, y=y, splitter=splitter) report_ridge_fe .. raw:: html
Pipeline(steps=[('functiontransformer',
                     FunctionTransformer(func=<function engineer_features at 0x7e0af04f8a90>)),
                    ('tablevectorizer', TableVectorizer()),
                    ('simpleimputer', SimpleImputer(add_indicator=True)),
                    ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                    ('ridge', Ridge())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 144-148 With this feature engineering, we improved the test score. However, SKD009 still flags the model because it cannot beat the HGB baseline on a majority of scores. .. GENERATED FROM PYTHON SOURCE LINES 148-151 .. code-block:: Python report_ridge_fe.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 152-158 Change model family: try a tree-based model =========================================== If nonlinearity and interactions matter, trees should close much of the gap. Let's compare a :class:`~sklearn.ensemble.RandomForestRegressor` pipeline to the engineered Ridge on the same split. .. GENERATED FROM PYTHON SOURCE LINES 158-177 .. code-block:: Python from sklearn.ensemble import RandomForestRegressor from skore import compare report_rf = evaluate( tabular_pipeline( RandomForestRegressor( n_estimators=100, min_samples_leaf=5, random_state=42, n_jobs=4, ) ), X=X, y=y, splitter=splitter, ) report_rf .. raw:: html
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                    unknown_value=-1))),
                    ('randomforestregressor',
                     RandomForestRegressor(min_samples_leaf=5, n_jobs=4,
                                           random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 178-186 .. code-block:: Python comparison_families = compare( { "ridge_with_fe": report_ridge_fe, "random_forest": report_rf, } ) comparison_families.metrics.summarize().frame() .. raw:: html
estimator ridge_with_fe random_forest
metric
default_score 0.519148 0.863328
r2 0.519148 0.863328
rmse 4317.071624 2301.565135
mae 3034.887147 1574.738667
mape 0.382612 0.175710
fit_time 0.450804 3.534885
predict_time 0.141589 0.174193


.. GENERATED FROM PYTHON SOURCE LINES 187-189 In this case, we see that we get closer to the gradient boosting baseline but we still have a small gap (SKD009 reports an r2 of around 0.88 for the baseline). .. GENERATED FROM PYTHON SOURCE LINES 189-192 .. code-block:: Python report_rf.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 193-199 Switch to HistGradientBoostingRegressor ======================================= skore's SKD009 performance baseline is itself an HGB pipeline. Matching that family is the natural next step once trees look promising, and we find that we are on par with skore's baseline. .. GENERATED FROM PYTHON SOURCE LINES 199-210 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingRegressor report_hgb = evaluate( tabular_pipeline(HistGradientBoostingRegressor(random_state=42)), X=X, y=y, splitter=splitter, ) report_hgb .. raw:: html
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingregressor',
                     HistGradientBoostingRegressor(random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 211-213 .. code-block:: Python report_hgb.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 214-230 Combine levers: features, HGB, and a log target =============================================== Let's go a little further in our modelling efforts in order to beat that performant baseline. For that, let us stack the previous techniques we used: - the engineered features, - an HGB with tuned parameters, - :class:`~sklearn.compose.TransformedTargetRegressor` with ``log1p`` / ``expm1``, because payment totals are heavy-tailed. In practice you would usually tune these knobs with :class:`~sklearn.model_selection.RandomizedSearchCV` or :class:`~sklearn.model_selection.GridSearchCV` from scikit-learn. To keep the example short and reproducible, we pin one search outcome that beats the baseline on this split. .. GENERATED FROM PYTHON SOURCE LINES 230-255 .. code-block:: Python from sklearn.compose import TransformedTargetRegressor from skrub import ToCategorical tuned = TransformedTargetRegressor( regressor=make_pipeline( FunctionTransformer(engineer_features), TableVectorizer(low_cardinality=ToCategorical()), HistGradientBoostingRegressor( learning_rate=0.05, max_iter=500, max_depth=5, max_leaf_nodes=63, min_samples_leaf=10, l2_regularization=0.1, random_state=42, ), ), func=np.log1p, inverse_func=np.expm1, ) report_tuned = evaluate(tuned, X=X, y=y, splitter=splitter) report_tuned .. raw:: html
TransformedTargetRegressor(func=<ufunc 'log1p'>, inverse_func=<ufunc 'expm1'>,
                               regressor=Pipeline(steps=[('functiontransformer',
                                                          FunctionTransformer(func=<function engineer_features at 0x7e0af04f8a90>)),
                                                         ('tablevectorizer',
                                                          TableVectorizer(low_cardinality=ToCategorical())),
                                                         ('histgradientboostingregressor',
                                                          HistGradientBoostingRegressor(l2_regularization=0.1,
                                                                                        learning_rate=0.05,
                                                                                        max_depth=5,
                                                                                        max_iter=500,
                                                                                        max_leaf_nodes=63,
                                                                                        min_samples_leaf=10,
                                                                                        random_state=42))]))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 256-258 Our model is now significantly better than an HGB baseline, and SKD009 is still reporting baseline scores for reference. .. GENERATED FROM PYTHON SOURCE LINES 258-261 .. code-block:: Python report_tuned.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 262-269 Conclusion ========== SKD009 compares a model with a HistGradientBoosting baseline. It warns when the model is significantly worse, and it still reports the baseline scores when the model is on par or better. The mitigations shown here are feature and target engineering, a change of model family, and hyperparameter tuning. .. rst-class:: sphx-glr-timing **Total running time of the script:** (2 minutes 26.441 seconds) .. _sphx_glr_download_auto_examples_pitfalls_and_solutions_plot_skd009_worse_than_baseline.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_skd009_worse_than_baseline.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_skd009_worse_than_baseline.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_skd009_worse_than_baseline.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_