SKD003 - Inconsistent performance across splits

SKD003 flags folds whose test metrics diverge sharply from the median during a cross-validation evaluation. With a proper splitter this is often a diagnostic check: the data have structure (groups, time, or a corrupted batch) that shuffled cross-validation would hide.

Realistic triggers:

• a contiguous bad batch of labels or features (mislabelled window, logging bug, schema mix-up),

• a much easier or harder group in one test fold under GroupKFold,

• temporal drift under TimeSeriesSplit (e.g. more ill patients start showing up),

• accidental fold imbalance from unshuffled KFold when prevalence varies along collection order (then shuffle or stratify if that will not appear in production).

When structure is real, shuffled cross-validation overestimates performance. SKD003 under a proper split is a good sign. The structure may not be fully fixable: once understood, mute with configuration() and consider collecting more data on the hard regime.

This notebook walks four beats: artificial corruption, a bad group in test, distribution shift in the last time-series fold, then ignoring SKD003 once the problem is understood.

Load Breast Cancer (two classes)

Let us use the Breast Cancer dataset to show how SKD003 can detect a batch of corrupted labels.

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer

cancer = load_breast_cancer(as_frame=True)
X, y = cancer.data, cancer.target

We can inspect the features and target with TableReport, and notice that it is a well curated dataset with two balanced classes (they appear in roughly equal proportions).

from skrub import TableReport

TableReport(X)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

TableReport(y)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Artificial corruption: a bad contiguous batch

Let us permute the labels on the first fifth of the dataset to break any association between X and y. Later, we will use unshuffled 5-fold cross-validation, so all corrupted rows will land in the same fold. We expect the score of this fold to be low due to this corruption. While we are creating this defect artificially, this is a scenario that can happen in practice due to e.g. a logging bug, a broken sensor, or a merge mix-up.

y_corrupted, n_corrupt = y.copy(), len(y) // 5
rng = np.random.default_rng(seed=0)
y_corrupted.iloc[:n_corrupt] = rng.permutation(y_corrupted.iloc[:n_corrupt])

We will use a default tabular_pipeline() classifier for preprocessing, and a gradient boosting model for prediction.

from sklearn.linear_model import LogisticRegression
from skrub import tabular_pipeline

model = tabular_pipeline(LogisticRegression())
model

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                ('logisticregression', LogisticRegression())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Trigger SKD003: cross-validate on corrupted labels

We will now evaluate the model on the corrupted labels using unshuffled 5-fold cross-validation and look at non aggregated metrics values, to notice the discrepancy on the first fold.

import skore

report = skore.evaluate(model, X=X, y=y_corrupted, pos_label=1, splitter=5)

report.metrics.summarize(data_source="test").frame(aggregate=None, flat_index=False)

estimator	LogisticRegression
split	Split #0	Split #1	Split #2	Split #3	Split #4
metric
accuracy	0.649123	0.859649	0.921053	0.964912	0.946903
precision	0.762712	0.816092	0.888889	0.947368	0.945205
recall	0.633803	1.000000	1.000000	1.000000	0.971831
roc_auc	0.640681	0.877825	0.994709	0.982474	0.989940
log_loss	1.688958	0.418982	0.228212	0.198544	0.217792
brier_score	0.309182	0.117186	0.060775	0.046597	0.054071
fit_time	0.148614	0.140205	0.146258	0.119379	0.120484
predict_time	0.083945	0.078751	0.067064	0.066841	0.065843

Looking at the checks results, SKD003 correctly flags split #0.

report.checks.summarize(fast_mode=True)

Visualize the variance across splits

The ROC display of a cross-validation report overlays the splits and summarizes them with a mean AUC and its standard deviation. It is built to show how much a model moves from one split to the next, rather than to identify a given split. Here the curves are widely spread and one of them sinks towards the chance level: this is the instability that SKD003 reported.

report.metrics.roc().plot()

<Figure size 600x750 with 1 Axes>

If the cause was a fixable bad batch, clean labels clear the outlier split.

report_clean = skore.evaluate(model, X=X, y=y, pos_label=1, splitter=5)
report_clean.metrics.summarize(data_source="test").frame(
    aggregate=None, flat_index=False
)

estimator	LogisticRegression
split	Split #0	Split #1	Split #2	Split #3	Split #4
metric
accuracy	0.991228	0.964912	0.991228	0.973684	0.991150
precision	0.986111	0.958904	0.986301	0.972603	1.000000
recall	1.000000	0.985915	1.000000	0.986111	0.985915
roc_auc	0.996725	0.994759	0.999008	0.988095	1.000000
log_loss	0.068116	0.088164	0.057122	0.098018	0.055142
brier_score	0.017134	0.028177	0.016488	0.021536	0.012032
fit_time	0.111174	0.113374	0.110567	0.111760	0.110291
predict_time	0.062849	0.063302	0.062700	0.062844	0.062409

report_clean.checks.summarize(fast_mode=True)

Bad group in the test fold

When observations carry a group identifier, e.g. the medical center where patient data were collected, a grouped splitter keeps each group on one side of every split. If one group is much harder or easier to predict, the fold that tests it will look like an outlier and SKD003 will fire. That is expected: the splitter did its job. Shuffled cross-validation would smear the difficult group across folds, hiding the gap and overestimating performance.

import skrub
from sklearn.model_selection import GroupKFold

n_batches = 10
batch_id = np.minimum(np.arange(len(X)) // (len(X) // n_batches), n_batches - 1)

y_batch = y.copy()
bad_batch = batch_id == 0
rng_batch = np.random.default_rng(seed=1)
y_batch.iloc[bad_batch] = rng_batch.choice(y.unique(), size=int(bad_batch.sum()))

df_batch = X.assign(batch_id=batch_id, target=y_batch)

GroupKFold needs the group vector at split time, so we use a skrub DataOp to attach it to the data. mark_as_X() accepts a cv argument and split_kwargs for group ids. The resulting learner carries its own cross-validation scheme, so evaluate() needs no splitter.

data = skrub.var("data", df_batch)
groups = data["batch_id"]
X_op = data.drop(columns=["batch_id", "target"]).skb.mark_as_X(
    cv=GroupKFold(n_splits=5),
    split_kwargs={"groups": groups},
)
y_op = data["target"].skb.mark_as_y()
learner = X_op.skb.apply(model, y=y_op).skb.make_learner()

report_grouped = skore.evaluate(learner, data={"data": df_batch}, pos_label=1)
report_grouped.metrics.summarize(data_source="test").frame(
    aggregate=None, flat_index=False
)

estimator	SkrubLearner
split	Split #0	Split #1	Split #2	Split #3	Split #4
metric
score	0.768595	0.955357	0.946429	0.973214	0.937500
accuracy	0.768595	0.955357	0.946429	0.973214	0.937500
precision	0.885246	0.948052	0.921053	0.962025	0.925926
recall	0.720000	0.986486	1.000000	1.000000	0.986842
roc_auc	0.811304	0.992888	0.993878	0.996345	0.978436
log_loss	1.552004	0.173870	0.171785	0.121842	0.173842
brier_score	0.212717	0.043370	0.047659	0.027993	0.046154
fit_time	0.149480	0.152578	0.196513	0.196123	0.194578
predict_time	0.093719	0.121204	0.122192	0.123455	0.124356

Looking at the checks results, SKD003 correctly flags split #0.

report_grouped.checks.summarize(fast_mode=True)

Distribution shift in the last time-series fold

Under TimeSeriesSplit, later windows can diverge from the training data. Consider a medical setting where diagnoses are collected over time: a sudden influx of ill patients near the end of the study shifts the class distribution. Earlier folds look strong because the model was trained on a balanced population, but the last fold drops as the positive class becomes rare. SKD003 fires, which is expected from chronological cross-validation and not a reason to reshuffle time.

We reuse the same breast-cancer dataset, attach a fake timestamp, and reduce the prevalence of the positive class in the last test window.

from sklearn.model_selection import TimeSeriesSplit

n_splits_time = 5
last_test_start = len(X) - (len(X) // (n_splits_time + 1))

y_time = y.copy()
rng_time = np.random.default_rng(seed=2)
y_time.iloc[last_test_start:] = rng_time.choice(
    [0, 1], size=len(X) - last_test_start, p=[0.95, 0.05]
)

timestamps = pd.date_range("2020-01-01", periods=len(X), freq="D")
df_time = X.assign(timestamp=timestamps, target=y_time)

As for the grouped section, we use a DataOp to declare the time-series split directly on the data.

data_time = skrub.var("data_time", df_time)
X_time_op = data_time.drop(columns=["timestamp", "target"]).skb.mark_as_X(
    cv=TimeSeriesSplit(n_splits=n_splits_time),
)
y_time_op = data_time["target"].skb.mark_as_y()
learner_time = X_time_op.skb.apply(model, y=y_time_op).skb.make_learner()

report_time = skore.evaluate(learner_time, data={"data_time": df_time}, pos_label=1)
report_time.metrics.summarize(data_source="test").frame(
    aggregate=None, flat_index=False
)

estimator	SkrubLearner
split	Split #0	Split #1	Split #2	Split #3	Split #4
metric
score	0.968085	0.957447	0.968085	0.978723	0.276596
accuracy	0.968085	0.957447	0.968085	0.978723	0.276596
precision	0.967213	0.977778	0.985507	1.000000	0.042857
recall	0.983333	0.936170	0.971429	0.972973	0.750000
roc_auc	0.994118	0.995926	0.986905	1.000000	0.397222
log_loss	0.103022	0.095178	0.090876	0.067273	3.835778
brier_score	0.027234	0.027583	0.021152	0.016549	0.664764
fit_time	0.196051	0.198472	0.174604	0.168912	0.200764
predict_time	0.130211	0.119480	0.095341	0.114247	0.118238

Split #4 underperforms because the positive class is now rare in that window. SKD003 here is expected from chronological cross-validation, not a reason to reshuffle time.

report_time.checks.summarize(fast_mode=True)

When the problem is understood: mute SKD003

Groups and drift are data properties: investigate, collect more labels on the hard regime if needed, but folds may never look uniform. Once SKD003 is expected, mute it with configuration() (or ignore=["SKD003"] on one summarize call).

with skore.configuration(ignore_checks=["SKD003"]):
    muted = report_time.checks.summarize(fast_mode=True)
muted

Side note: if there is no group or time structure, but an unshuffled KFold still creates imbalanced folds because class prevalence varies along the collection order (for example a sensor failed for part of the dump), shuffling or stratifying is appropriate when you are sure that irregularity is accidental and will not appear in production. That case is the exception where changing the splitter to smooth folds is the right fix; do not use it to hide real groups or time drift.

Conclusion

SKD003 is a reminder to inspect unstable cross-validation folds. With a proper splitter, firing often means the evaluation exposed a bad batch, a hard group, or temporal shift. Fix what you can (for example a corrupted label window). When the structure is intrinsic, keep the honest splitter, document the outlier regime, mute SKD003 via configuration, and collect more data on that regime if you need better coverage. Avoid shuffled cross-validation as a way to make the check disappear when groups or time are real. See also SKD013 for chronological train/test overlap on hold-out reports.