.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/pitfalls_and_solutions/plot_skd016_estimator_not_tuned.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_pitfalls_and_solutions_plot_skd016_estimator_not_tuned.py: .. _example_skd016_estimator_not_tuned: SKD016 - Estimator not tuned ============================ This example walks through mitigations when check :ref:`SKD016 ` tips on a plain estimator left at scikit-learn defaults. The check compares initialization parameters against a curated table of high-impact hyperparameters and suggests axes worth tuning. Mitigations from the :ref:`automated_checks` user guide: - wrap the estimator in :class:`~sklearn.model_selection.GridSearchCV` or :class:`~sklearn.model_selection.RandomizedSearchCV` over the suggested parameters, - or set sensible non-default values manually. We use the employee salaries dataset (above-median salary as the positive class) with a default :func:`~skrub.tabular_pipeline` classifier. The goal is to move off factory defaults either through search or hand-picked values. .. GENERATED FROM PYTHON SOURCE LINES 25-32 Load the employee salaries dataset ================================== Mixed HR features suit ``tabular_pipeline``. A default ``tabular_pipeline("classifier")`` leaves :class:`~sklearn.ensemble.HistGradientBoostingClassifier` at sklearn defaults — the setup SKD016 is designed to flag. .. GENERATED FROM PYTHON SOURCE LINES 32-40 .. code-block:: Python from skrub.datasets import fetch_employee_salaries dataset = fetch_employee_salaries() X = dataset.X y_salary = dataset.y.squeeze() y = (y_salary > y_salary.median()).astype(int).rename("high_earner") .. GENERATED FROM PYTHON SOURCE LINES 41-42 Inspect inputs and the binary target with :class:`~skrub.TableReport`. .. GENERATED FROM PYTHON SOURCE LINES 42-47 .. code-block:: Python from skrub import TableReport TableReport(X) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 48-50 .. code-block:: Python TableReport(y) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 51-52 We use the same stratified split for every comparison. .. GENERATED FROM PYTHON SOURCE LINES 52-57 .. code-block:: Python from skore import TrainTestSplit splitter = TrainTestSplit(test_size=0.2, random_state=42, stratify=y) .. GENERATED FROM PYTHON SOURCE LINES 58-64 Trigger SKD016 - untuned default pipeline ========================================= Defaults are fine for a first look at the table, but they are not a production configuration. SKD016 names the high-impact axes that usually matter first for this estimator family. .. GENERATED FROM PYTHON SOURCE LINES 64-77 .. code-block:: Python import skore from skrub import tabular_pipeline report = skore.evaluate( tabular_pipeline("classifier"), X=X, y=y, pos_label=1, splitter=splitter, ) report .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingclassifier',
                     HistGradientBoostingClassifier())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 78-80 SKD016 should tip that HistGradientBoostingClassifier remains at defaults. Read which parameters it lists — the next sections search or set those knobs. .. GENERATED FROM PYTHON SOURCE LINES 80-83 .. code-block:: Python report.checks.summarize() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 84-86 .. code-block:: Python report.metrics.summarize(data_source="both").frame() .. raw:: html
HistGradientBoostingClassifier (train) HistGradientBoostingClassifier (test)
metric
accuracy 0.988756 0.937161
precision 0.992087 0.938111
recall 0.985366 0.936078
roc_auc 0.999151 0.987118
log_loss 0.054099 0.146457
brier_score 0.012345 0.044194
fit_time 2.044371 2.044371
predict_time 0.782540 0.284502


.. GENERATED FROM PYTHON SOURCE LINES 87-97 Wrap the estimator in RandomizedSearchCV ======================================== Search the axes SKD016 typically flags for HGB (learning rate, iteration budget, depth, leaf size) instead of accepting sklearn defaults. Once the report wraps a fitted search object, SKD016 clears. A tuned search can still raise :ref:`SKD014 ` or :ref:`SKD015 ` if the box is too narrow or incomplete — see that combined example. .. GENERATED FROM PYTHON SOURCE LINES 97-130 .. code-block:: Python from scipy.stats import loguniform, randint from sklearn.ensemble import HistGradientBoostingClassifier from sklearn.model_selection import RandomizedSearchCV base_pipeline = tabular_pipeline(HistGradientBoostingClassifier(random_state=42)) param_distributions = { "histgradientboostingclassifier__learning_rate": loguniform(1e-2, 2e-1), "histgradientboostingclassifier__max_iter": randint(100, 401), "histgradientboostingclassifier__max_depth": [3, 5, 8, None], "histgradientboostingclassifier__min_samples_leaf": randint(10, 51), } tuned_search = RandomizedSearchCV( base_pipeline, param_distributions=param_distributions, n_iter=8, cv=3, scoring="neg_log_loss", random_state=42, refit=True, ) report_tuned = skore.evaluate( tuned_search, X=X, y=y, pos_label=1, splitter=splitter, ) report_tuned .. raw:: html 
RandomizedSearchCV(cv=3,
                       estimator=Pipeline(steps=[('tablevectorizer',
                                                  TableVectorizer(low_cardinality=ToCategorical())),
                                                 ('histgradientboostingclassifier',
                                                  HistGradientBoostingClassifier(random_state=42))]),
                       n_iter=8,
                       param_distributions={'histgradientboostingclassifier__learning_rate': <scipy.stats._distn_infrastructure.rv_continuous_frozen object at 0x7f7...9e0>,
                                            'histgradientboostingclassifier__max_depth': [3,
                                                                                          5,
                                                                                          8,
                                                                                          None],
                                            'histgradientboostingclassifier__max_iter': <scipy.stats._distn_infrastructure.rv_discrete_frozen object at 0x7f782c6387c0>,
                                            'histgradientboostingclassifier__min_samples_leaf': <scipy.stats._distn_infrastructure.rv_discrete_frozen object at 0x7f782c639bf0>},
                       random_state=42, scoring='neg_log_loss')
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 131-132 SKD016 should be absent; the report wraps a fitted search object. .. GENERATED FROM PYTHON SOURCE LINES 132-135 .. code-block:: Python report_tuned.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 136-138 .. code-block:: Python report_tuned.estimator_.best_params_ .. rst-class:: sphx-glr-script-out .. code-block:: none {'histgradientboostingclassifier__learning_rate': np.float64(0.060099747183803134), 'histgradientboostingclassifier__max_depth': 8, 'histgradientboostingclassifier__max_iter': 221, 'histgradientboostingclassifier__min_samples_leaf': 28} .. GENERATED FROM PYTHON SOURCE LINES 139-146 Set sensible non-default values manually ======================================== When a full search is impractical, hand-pick hyperparameters that differ from defaults. SKD016 clears as soon as impactful knobs are no longer factory settings — that is an intentional configuration signal, not proof that the values are optimal. Prefer validated search when you can afford it. .. GENERATED FROM PYTHON SOURCE LINES 146-166 .. code-block:: Python model_manual = tabular_pipeline( HistGradientBoostingClassifier( learning_rate=0.05, max_iter=200, max_depth=5, min_samples_leaf=20, random_state=42, ) ) report_manual = skore.evaluate( model_manual, X=X, y=y, pos_label=1, splitter=splitter, ) report_manual .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingclassifier',
                     HistGradientBoostingClassifier(learning_rate=0.05, max_depth=5,
                                                    max_iter=200,
                                                    random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 167-168 SKD016 should be absent once hyperparameters differ from defaults. .. GENERATED FROM PYTHON SOURCE LINES 168-171 .. code-block:: Python report_manual.checks.summarize(fast_mode=True) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 172-174 .. code-block:: Python report_manual.metrics.summarize(data_source="both").frame() .. raw:: html
HistGradientBoostingClassifier (train) HistGradientBoostingClassifier (test)
metric
accuracy 0.969927 0.933369
precision 0.968649 0.931034
recall 0.971274 0.936078
roc_auc 0.996075 0.985018
log_loss 0.096833 0.157725
brier_score 0.025237 0.046415
fit_time 1.383695 1.383695
predict_time 0.469582 0.155745


.. GENERATED FROM PYTHON SOURCE LINES 175-181 Compare mitigations =================== Hold-out metrics for the default pipeline and the hand-tuned one. The RandomizedSearchCV report wraps a search estimator, so its metric table can look different in :func:`~skore.compare`; we show it in its own cell below. .. GENERATED FROM PYTHON SOURCE LINES 181-190 .. code-block:: Python comparison = skore.compare( { "default_pipeline": report, "hand_tuned_hgb": report_manual, } ) comparison.metrics.summarize(data_source="both").frame() .. raw:: html
default_pipeline (train) default_pipeline (test) hand_tuned_hgb (train) hand_tuned_hgb (test)
metric
accuracy 0.988756 0.937161 0.969927 0.933369
precision 0.992087 0.938111 0.968649 0.931034
recall 0.985366 0.936078 0.971274 0.936078
roc_auc 0.999151 0.987118 0.996075 0.985018
log_loss 0.054099 0.146457 0.096833 0.157725
brier_score 0.012345 0.044194 0.025237 0.046415
fit_time 2.044371 2.044371 1.383695 1.383695
predict_time 0.782540 0.284502 0.469582 0.155745


.. GENERATED FROM PYTHON SOURCE LINES 191-193 .. code-block:: Python report_tuned.metrics.summarize(data_source="both").frame() .. raw:: html
RandomizedSearchCV (train) RandomizedSearchCV (test)
metric
score -0.048947 -0.150687
accuracy 0.988756 0.937703
precision 0.991551 0.937229
recall 0.985908 0.938245
roc_auc 0.999260 0.986551
log_loss 0.048947 0.150687
brier_score 0.011166 0.045203
fit_time 47.923457 47.923457
predict_time 0.490584 0.166994


.. GENERATED FROM PYTHON SOURCE LINES 194-202 Conclusion ========== SKD016 nudges you off scikit-learn defaults for high-impact estimators. Randomized search and hand-tuned HGB parameters both clear the tip; clearing the check means you left factory settings, not that the model is finished. Pair manual choices with periodic search, and watch SKD014/SKD015 once you wrap a ``BaseSearchCV``. .. rst-class:: sphx-glr-timing **Total running time of the script:** (1 minutes 42.095 seconds) .. _sphx_glr_download_auto_examples_pitfalls_and_solutions_plot_skd016_estimator_not_tuned.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_skd016_estimator_not_tuned.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_skd016_estimator_not_tuned.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_skd016_estimator_not_tuned.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_