.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/technical_details/plot_diagnostic_api.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_technical_details_plot_diagnostic_api.py: .. _example_diagnostic_api: ======================================= Automatic detection of modelling issues ======================================= `skore` can automatically detect common modeling pitfalls such as overfitting and underfitting. This example walks through the ``.diagnose`` method: how to run checks, how to read the detected issues, and how to mute specific checks. We use a purely non-linear regression target and deliberately pick models that fail in known ways: - a **linear model** that cannot capture the non-linearity → underfitting, - a **single deep decision tree** that memorizes the training set perfectly and fails to generalize → overfitting. .. GENERATED FROM PYTHON SOURCE LINES 21-27 Setup ===== The target is a product of trigonometric functions of the first two features: completely invisible to a linear model, yet easy to memorize for an unconstrained tree. .. GENERATED FROM PYTHON SOURCE LINES 27-42 .. code-block:: Python import numpy as np from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor rng = np.random.default_rng(42) n_samples = 500 X = rng.uniform(0, 1, (n_samples, 5)) y = np.sin(2 * np.pi * X[:, 0]) * np.cos(2 * np.pi * X[:, 1]) + rng.normal( 0, 0.1, n_samples ) linear = LinearRegression() deep_tree = DecisionTreeRegressor(random_state=42) .. GENERATED FROM PYTHON SOURCE LINES 43-49 Calling :meth:`~skore.EstimatorReport.diagnose` explicitly ========================================================== Every report exposes a :meth:`~skore.EstimatorReport.diagnose` method. Checks are computed lazily and cached, so calling :meth:`~skore.EstimatorReport.diagnose` is always cheap after the first call. .. GENERATED FROM PYTHON SOURCE LINES 49-55 .. code-block:: Python from skore import evaluate linear_report = evaluate(linear, X, y) linear_report .. raw:: html 
LinearRegression()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

1 issue(s), 1 tip(s), 1 passed, 0 ignored.


.. GENERATED FROM PYTHON SOURCE LINES 56-58 .. code-block:: Python linear_report.diagnose() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 59-61 .. code-block:: Python linear_report.metrics.summarize(data_source="both").frame() .. raw:: html
LinearRegression (train) LinearRegression (test)
Metric
R² 0.001906 -0.015818
RMSE 0.522214 0.504156
MAE 0.423723 0.406739
MAPE 1.426344 1.032250
Fit time (s) 0.000992 0.000992
Predict time (s) 0.000137 0.000304


.. GENERATED FROM PYTHON SOURCE LINES 62-64 The linear model is flagged for underfitting: its scores are on par between train and test, and not significantly better than a dummy baseline. .. GENERATED FROM PYTHON SOURCE LINES 64-68 .. code-block:: Python tree_report = evaluate(deep_tree, X, y) tree_report.diagnose() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 69-71 .. code-block:: Python tree_report.metrics.summarize(data_source="both").frame() .. raw:: html
DecisionTreeRegressor (train) DecisionTreeRegressor (test)
Metric
R² 1.000000 0.783887
RMSE 0.000000 0.232540
MAE 0.000000 0.180261
MAPE 0.000000 1.052768
Fit time (s) 0.002861 0.002861
Predict time (s) 0.000264 0.000236


.. GENERATED FROM PYTHON SOURCE LINES 72-74 The deep tree is flagged for overfitting: it achieves a perfect score on train but degrades on test. .. GENERATED FROM PYTHON SOURCE LINES 76-81 Ignoring specific checks ======================== Each check has a stable code (e.g. ``SKD001``, ``SKD002``). You can mute individual checks per call: .. GENERATED FROM PYTHON SOURCE LINES 81-84 .. code-block:: Python tree_report.diagnose(ignore=["SKD001"]) .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 85-87 Or globally, so that every subsequent :meth:`~skore.EstimatorReport.diagnose` call skips them: .. GENERATED FROM PYTHON SOURCE LINES 87-94 .. code-block:: Python import skore with skore.configuration(ignore_checks=["SKD001"]): diagnosis = tree_report.diagnose() diagnosis .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 95-100 Diagnostics on a :class:`~skore.CrossValidationReport` ====================================================== When ``splitter`` is an integer, :func:`~skore.evaluate` returns a :class:`~skore.CrossValidationReport`. Checks aggregate issues across folds. .. GENERATED FROM PYTHON SOURCE LINES 100-104 .. code-block:: Python cv_report = evaluate(deep_tree, X, y, splitter=5) cv_report.diagnose() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 105-110 Diagnostics on a :class:`~skore.ComparisonReport` ================================================= Passing a list of estimators returns a :class:`~skore.ComparisonReport`. Issues are grouped by sub-report. .. GENERATED FROM PYTHON SOURCE LINES 110-113 .. code-block:: Python comparison_report = evaluate([linear, deep_tree], X, y) comparison_report.diagnose() .. raw:: html


.. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 0.816 seconds) .. _sphx_glr_download_auto_examples_technical_details_plot_diagnostic_api.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_diagnostic_api.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_diagnostic_api.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_diagnostic_api.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_