Automatic detection of modelling issues

skore can automatically detect common modeling pitfalls such as overfitting and underfitting. This example walks through the .diagnose method: how to run checks, how to read the detected issues, and how to mute specific checks.

We use a purely non-linear regression target and deliberately pick models that fail in known ways:

• a linear model that cannot capture the non-linearity → underfitting,

• a single deep decision tree that memorizes the training set perfectly and fails to generalize → overfitting.

Setup

The target is a product of trigonometric functions of the first two features: completely invisible to a linear model, yet easy to memorize for an unconstrained tree.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(42)
n_samples = 500
X = rng.uniform(0, 1, (n_samples, 5))
y = np.sin(2 * np.pi * X[:, 0]) * np.cos(2 * np.pi * X[:, 1]) + rng.normal(
    0, 0.1, n_samples
)

linear = LinearRegression()
deep_tree = DecisionTreeRegressor(random_state=42)

Calling diagnose() explicitly

Every report exposes a diagnose() method. Checks are computed lazily and cached, so calling diagnose() is always cheap after the first call.

from skore import evaluate

linear_report = evaluate(linear, X, y)
linear_report

LinearRegression()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

1 issue(s) detected, 2 check(s) ran.

linear_report.diagnose()

Diagnostic: 1 issue(s) detected, 2 check(s) ran, 0 ignored.

• [SKD002] Potential underfitting. Train/test scores are on par and not significantly better than the dummy baseline for 2/2 comparable metrics. Read more about this here. Mute with ignore=['SKD002'].

linear_report.metrics.summarize(data_source="both").frame()

	LinearRegression (train)	LinearRegression (test)
Metric
R²	0.001906	-0.015818
RMSE	0.522214	0.504156
Fit time (s)	0.001152	0.001152
Predict time (s)	0.000152	0.000292

The linear model is flagged for underfitting: its scores are on par between train and test, and not significantly better than a dummy baseline.

tree_report = evaluate(deep_tree, X, y)
tree_report.diagnose()

Diagnostic: 1 issue(s) detected, 2 check(s) ran, 0 ignored.

• [SKD001] Potential overfitting. Significant train/test gaps were found for 2/2 default predictive metrics. Read more about this here. Mute with ignore=['SKD001'].

tree_report.metrics.summarize(data_source="both").frame()

	DecisionTreeRegressor (train)	DecisionTreeRegressor (test)
Metric
R²	1.000000	0.783887
RMSE	0.000000	0.232540
Fit time (s)	0.003070	0.003070
Predict time (s)	0.000221	0.000195

The deep tree is flagged for overfitting: it achieves a perfect score on train but degrades on test.

Ignoring specific checks

Each check has a stable code (e.g. SKD001, SKD002). You can mute individual checks per call:

tree_report.diagnose(ignore=["SKD001"])

Diagnostic: 0 issue(s) detected, 1 check(s) ran, 1 ignored.

• No issues were detected in your report!

Or globally, so that every subsequent diagnose() call skips them:

import skore

with skore.configuration(ignore_checks=["SKD001"]):
    diagnosis = tree_report.diagnose()
diagnosis

Diagnostic: 0 issue(s) detected, 1 check(s) ran, 1 ignored.

• No issues were detected in your report!

Diagnostics on a CrossValidationReport

When splitter is an integer, evaluate() returns a CrossValidationReport. Checks aggregate issues across folds.

cv_report = evaluate(deep_tree, X, y, splitter=5)
cv_report.diagnose()

Diagnostic: 1 issue(s) detected, 2 check(s) ran, 0 ignored.

• [SKD001] Potential overfitting. Detected in 5/5 evaluated splits. Read more about this here. Mute with ignore=['SKD001'].

Diagnostics on a ComparisonReport

Passing a list of estimators returns a ComparisonReport. Issues are grouped by sub-report.

comparison_report = evaluate([linear, deep_tree], X, y)
comparison_report.diagnose()

Diagnostic: 2 issue(s) detected, 2 check(s) ran, 0 ignored.

• [SKD002] Potential underfitting. [LinearRegression] Train/test scores are on par and not significantly better than the dummy baseline for 2/2 comparable metrics. Read more about this here. Mute with ignore=['SKD002'].
• [SKD001] Potential overfitting. [DecisionTreeRegressor] Significant train/test gaps were found for 2/2 default predictive metrics. Read more about this here. Mute with ignore=['SKD001'].