.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/pitfalls_and_solutions/plot_skd009_worse_than_baseline.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_pitfalls_and_solutions_plot_skd009_worse_than_baseline.py: .. _example_skd009_worse_than_baseline: SKD009 - Model worse than baseline ================================== This example walks through mitigations when check :ref:`SKD009 ` fires. The check trains a strong :func:`~skrub.tabular_pipeline` baseline (gradient boosting on vectorized data) and flags estimators that are significantly worse on default metrics. Mitigations from the :ref:`user guide `, in the order we try them here: - revisit feature engineering and preprocessing, - check whether the model family is appropriate, - switch to a stronger default such as HistGradientBoostingRegressor, - tune the model (here: moderated HGB capacity plus a log target). We use the medical charge dataset with provider IDs and leakage columns removed. We will see how applying these recommendations leads to a model that improves on the baseline. .. GENERATED FROM PYTHON SOURCE LINES 26-32 Load the medical charge dataset =============================== :func:`skrub.datasets.fetch_medical_charge` returns hospital billing records. We drop provider identifiers and columns that leak the target, then subsample 2,000 rows for a challenging regression task. .. GENERATED FROM PYTHON SOURCE LINES 32-49 .. code-block:: Python from skrub.datasets import fetch_medical_charge dataset = fetch_medical_charge() X_full, y_full = dataset.X, dataset.y id_cols = [ "Provider_Zip_Code", "Provider_Id", "Provider_Name", "Provider_Street_Address", ] leakage_cols = ["Average_Covered_Charges", "Average_Medicare_Payments"] X = X_full.drop(columns=id_cols + leakage_cols).sample(2_000, random_state=42) y = y_full.loc[X.index] .. rst-class:: sphx-glr-script-out .. code-block:: none Downloading 'medical_charge' from https://github.com/skrub-data/skrub-data-files/raw/refs/heads/main/medical_charge.zip (attempt 1/3) .. GENERATED FROM PYTHON SOURCE LINES 50-51 Inspect the features matrix with :class:`~skrub.TableReport`. .. GENERATED FROM PYTHON SOURCE LINES 51-56 .. code-block:: Python from skrub import TableReport TableReport(X) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 57-58 The target is continuous total payment. .. GENERATED FROM PYTHON SOURCE LINES 58-61 .. code-block:: Python TableReport(y) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 62-66 .. code-block:: Python from skore import TrainTestSplit splitter = TrainTestSplit(random_state=42, test_size=0.2) .. GENERATED FROM PYTHON SOURCE LINES 67-73 Trigger SKD009 with a linear pipeline ===================================== Start with :func:`~skrub.tabular_pipeline` around :class:`~sklearn.linear_model.Ridge` so that encoding and imputation are already in place before fitting the linear model. .. GENERATED FROM PYTHON SOURCE LINES 73-81 .. code-block:: Python from sklearn.linear_model import Ridge from skore import evaluate from skrub import tabular_pipeline report_ridge = evaluate(tabular_pipeline(Ridge()), X=X, y=y, splitter=splitter) report_ridge .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                    ('simpleimputer', SimpleImputer(add_indicator=True)),
                    ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                    ('ridge', Ridge())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 82-84 Find ``SKD009`` in the Tips tab below: the Ridge pipeline reports worse-than-baseline performance on a majority of metrics. .. GENERATED FROM PYTHON SOURCE LINES 84-87 .. code-block:: Python report_ridge.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 88-107 Revisit feature engineering =========================== A linear model benefits from spelling out structure that trees would discover from the raw columns: - ``log1p(Total_Discharges)`` compresses a skewed volume signal (then drop the raw count to avoid collinearity), - extract the numeric DRG code from labels like ``"178 - ... W CC"``, - flag severity markers in the text (``W MCC`` / ``W CC``), which affect payment levels. Keep everything inside a pipeline so the same transforms run at predict time on new data. Since this transformation is stateless, i.e., no information needs to be computed and stored during training, we can safely use a :class:`~sklearn.preprocessing.FunctionTransformer`. We also break down the :func:`~skrub.tabular_pipeline` into its components (vectorization of data, missing value imputation and scaling) to keep the :class:`~sklearn.pipeline.Pipeline` flat. .. GENERATED FROM PYTHON SOURCE LINES 107-142 .. code-block:: Python import numpy as np from sklearn.impute import SimpleImputer from sklearn.pipeline import make_pipeline from sklearn.preprocessing import FunctionTransformer from skrub import SquashingScaler, TableVectorizer def engineer_features(X): """Stateless feature engineering.""" drg = X["DRG_Definition"].str.upper() return X.drop(columns=["Total_Discharges"]).assign( log_Total_Discharges=np.log1p(X["Total_Discharges"]), DRG_Code=( X["DRG_Definition"].str.extract(r"^(\d+)", expand=False).astype(float) ), has_MCC=drg.str.contains("W MCC", regex=False).astype(int), has_CC=( drg.str.contains("W CC", regex=False) & ~drg.str.contains("W MCC", regex=False) ).astype(int), ) ridge_with_fe = make_pipeline( FunctionTransformer(engineer_features), TableVectorizer(), SimpleImputer(add_indicator=True), SquashingScaler(max_absolute_value=5), Ridge(), ) report_ridge_fe = evaluate(ridge_with_fe, X=X, y=y, splitter=splitter) report_ridge_fe .. raw:: html 
Pipeline(steps=[('functiontransformer',
                     FunctionTransformer(func=<function engineer_features at 0x7f26d02c2770>)),
                    ('tablevectorizer', TableVectorizer()),
                    ('simpleimputer', SimpleImputer(add_indicator=True)),
                    ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                    ('ridge', Ridge())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 143-147 With this feature engineering, we improved the test score. However, SKD009 still flags the model because it cannot beat the HGB baseline on a majority of scores. .. GENERATED FROM PYTHON SOURCE LINES 147-150 .. code-block:: Python report_ridge_fe.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 151-157 Change model family: try a tree-based model =========================================== If nonlinearity and interactions matter, trees should close much of the gap. Let's compare a :class:`~sklearn.ensemble.RandomForestRegressor` pipeline to the engineered Ridge on the same split. .. GENERATED FROM PYTHON SOURCE LINES 157-176 .. code-block:: Python from sklearn.ensemble import RandomForestRegressor from skore import compare report_rf = evaluate( tabular_pipeline( RandomForestRegressor( n_estimators=100, min_samples_leaf=5, random_state=42, n_jobs=4, ) ), X=X, y=y, splitter=splitter, ) report_rf .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                    unknown_value=-1))),
                    ('randomforestregressor',
                     RandomForestRegressor(min_samples_leaf=5, n_jobs=4,
                                           random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 177-185 .. code-block:: Python comparison_families = compare( { "ridge_with_fe": report_ridge_fe, "random_forest": report_rf, } ) comparison_families.metrics.summarize().frame() .. raw:: html
estimator ridge_with_fe random_forest
metric
r2 0.505097 0.861134
rmse 4379.691473 2319.966730
mae 3026.040558 1598.113803
mape 0.383160 0.177348
fit_time 0.565683 3.612842
predict_time 0.119716 0.139302


.. GENERATED FROM PYTHON SOURCE LINES 186-188 In this case, we see that we get closer to the gradient boosting baseline but we still have a small gap (SKD009 reports an r2 of around 0.88 for the baseline). .. GENERATED FROM PYTHON SOURCE LINES 188-191 .. code-block:: Python report_rf.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 192-198 Switch to HistGradientBoostingRegressor ======================================= skore's SKD009 performance baseline is itself an HGB pipeline. Matching that family is the natural next step once trees look promising, and we find that we are on par with skore's baseline. .. GENERATED FROM PYTHON SOURCE LINES 198-209 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingRegressor report_hgb = evaluate( tabular_pipeline(HistGradientBoostingRegressor(random_state=42)), X=X, y=y, splitter=splitter, ) report_hgb .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingregressor',
                     HistGradientBoostingRegressor(random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 210-212 .. code-block:: Python report_hgb.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 213-229 Combine levers: features, HGB, and a log target =============================================== Let's go a little further in our modelling efforts in order to beat that performant baseline. For that, let us stack the previous techniques we used: - the engineered features, - an HGB with tuned parameters, - :class:`~sklearn.compose.TransformedTargetRegressor` with ``log1p`` / ``expm1``, because payment totals are heavy-tailed. In practice you would usually tune these knobs with :class:`~sklearn.model_selection.RandomizedSearchCV` or :class:`~sklearn.model_selection.GridSearchCV` from scikit-learn. To keep the example short and reproducible, we pin one search outcome that beats the baseline on this split. .. GENERATED FROM PYTHON SOURCE LINES 229-254 .. code-block:: Python from sklearn.compose import TransformedTargetRegressor from skrub import ToCategorical tuned = TransformedTargetRegressor( regressor=make_pipeline( FunctionTransformer(engineer_features), TableVectorizer(low_cardinality=ToCategorical()), HistGradientBoostingRegressor( learning_rate=0.05, max_iter=500, max_depth=5, max_leaf_nodes=63, min_samples_leaf=10, l2_regularization=0.1, random_state=42, ), ), func=np.log1p, inverse_func=np.expm1, ) report_tuned = evaluate(tuned, X=X, y=y, splitter=splitter) report_tuned .. raw:: html 
TransformedTargetRegressor(func=<ufunc 'log1p'>, inverse_func=<ufunc 'expm1'>,
                               regressor=Pipeline(steps=[('functiontransformer',
                                                          FunctionTransformer(func=<function engineer_features at 0x7f26d02c2770>)),
                                                         ('tablevectorizer',
                                                          TableVectorizer(low_cardinality=ToCategorical())),
                                                         ('histgradientboostingregressor',
                                                          HistGradientBoostingRegressor(l2_regularization=0.1,
                                                                                        learning_rate=0.05,
                                                                                        max_depth=5,
                                                                                        max_iter=500,
                                                                                        max_leaf_nodes=63,
                                                                                        min_samples_leaf=10,
                                                                                        random_state=42))]))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 255-257 Our model is now significantly better than an HGB baseline, and SKD009 is still reporting baseline scores for reference. .. GENERATED FROM PYTHON SOURCE LINES 257-260 .. code-block:: Python report_tuned.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 261-268 Conclusion ========== SKD009 informs you when your model is beaten by a simple but strong baseline. We showed different approaches to address it: feature and target engineering, changing model family and tuning hyperparameters to finally clear well past the baseline. .. rst-class:: sphx-glr-timing **Total running time of the script:** (1 minutes 54.319 seconds) .. _sphx_glr_download_auto_examples_pitfalls_and_solutions_plot_skd009_worse_than_baseline.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_skd009_worse_than_baseline.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_skd009_worse_than_baseline.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_skd009_worse_than_baseline.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_