{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n\n# SKD003 - Inconsistent performance across splits\n\n`SKD003 ` flags folds whose test metrics diverge\nsharply from the median during a cross-validation evaluation. With a proper splitter\nthis is often a diagnostic check: the data have structure (groups, time, or a corrupted\nbatch) that shuffled cross-validation would hide.\n\nRealistic triggers:\n\n- a contiguous bad batch of labels or features (mislabelled window, logging\n bug, schema mix-up),\n- a much easier or harder group in one test fold under\n :class:`~sklearn.model_selection.GroupKFold`,\n- temporal drift under :class:`~sklearn.model_selection.TimeSeriesSplit`\n (e.g. more ill patients start showing up),\n- accidental fold imbalance from unshuffled\n :class:`~sklearn.model_selection.KFold` when prevalence varies along\n collection order (then shuffle or stratify if that will not appear in\n production).\n\nWhen structure is real, shuffled cross-validation overestimates performance. SKD003\nunder a proper split is a good sign. The structure may not be fully fixable: once\nunderstood, mute with :func:`~skore.configuration` and consider collecting more data on\nthe hard regime.\n\nThis notebook walks four beats: artificial corruption, a bad group in test, distribution\nshift in the last time-series fold, then ignoring SKD003 once the problem is understood.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load Breast Cancer (two classes)\n\nLet us use the Breast Cancer dataset to show how SKD003 can detect a batch of\ncorrupted labels.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import numpy as np\nimport pandas as pd\nfrom sklearn.datasets import load_breast_cancer\n\ncancer = load_breast_cancer(as_frame=True)\nX, y = cancer.data, cancer.target" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can inspect the features and target with :class:`~skrub.TableReport`, and\nnotice that it is a well curated dataset with two balanced classes (they\nappear in roughly equal proportions).\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from skrub import TableReport\n\nTableReport(X)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "TableReport(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Artificial corruption: a bad contiguous batch\n\nLet us permute the labels on the first fifth of the dataset to break any\nassociation between `X` and `y`. Later, we will use unshuffled 5-fold\ncross-validation, so all corrupted rows will land in the same fold. We expect\nthe score of this fold to be low due to this corruption. While we are\ncreating this defect artificially, this is a scenario that can happen in\npractice due to e.g. a logging bug, a broken sensor, or a merge mix-up.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "y_corrupted, n_corrupt = y.copy(), len(y) // 5\nrng = np.random.default_rng(seed=0)\ny_corrupted.iloc[:n_corrupt] = rng.permutation(y_corrupted.iloc[:n_corrupt])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will use a default :func:`~skrub.tabular_pipeline` classifier for preprocessing, and\na gradient boosting model for prediction.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.linear_model import LogisticRegression\nfrom skrub import tabular_pipeline\n\nmodel = tabular_pipeline(LogisticRegression())\nmodel" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Trigger SKD003: cross-validate on corrupted labels\n\nWe will now evaluate the model on the corrupted labels using unshuffled\n5-fold cross-validation and look at non aggregated metrics values, to notice\nthe discrepancy on the first fold.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import skore\n\nreport = skore.evaluate(model, X=X, y=y_corrupted, pos_label=1, splitter=5)\n\nreport.metrics.summarize(data_source=\"test\").frame(aggregate=None, flat_index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking at the checks results, ``SKD003`` correctly flags split #0.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Visualize the variance across splits\n\nThe ROC display of a cross-validation report overlays the splits and summarizes them\nwith a mean AUC and its standard deviation. It is built to show how much a model moves\nfrom one split to the next, rather than to identify a given split. Here the curves are\nwidely spread and one of them sinks towards the chance level: this is the instability\nthat ``SKD003`` reported.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.metrics.roc().plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the cause was a fixable bad batch, clean labels clear the outlier split.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_clean = skore.evaluate(model, X=X, y=y, pos_label=1, splitter=5)\nreport_clean.metrics.summarize(data_source=\"test\").frame(\n aggregate=None, flat_index=False\n)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_clean.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Bad group in the test fold\n\nWhen observations carry a group identifier, e.g. the medical center where\npatient data were collected, a grouped splitter keeps each group on one side of\nevery split. If one group is much harder or easier to predict, the fold that tests\nit will look like an outlier and ``SKD003`` will fire. That is expected: the splitter\ndid its job. Shuffled cross-validation would smear the difficult group across folds,\nhiding the gap and overestimating performance.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import skrub\nfrom sklearn.model_selection import GroupKFold\n\nn_batches = 10\nbatch_id = np.minimum(np.arange(len(X)) // (len(X) // n_batches), n_batches - 1)\n\ny_batch = y.copy()\nbad_batch = batch_id == 0\nrng_batch = np.random.default_rng(seed=1)\ny_batch.iloc[bad_batch] = rng_batch.choice(y.unique(), size=int(bad_batch.sum()))\n\ndf_batch = X.assign(batch_id=batch_id, target=y_batch)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":class:`~sklearn.model_selection.GroupKFold` needs the group vector at split time,\nso we use a skrub :class:`~skrub.DataOp` to attach it to the data.\n:meth:`~skrub.DataOp.skb.mark_as_X` accepts a ``cv`` argument and\n``split_kwargs`` for group ids. The resulting learner carries its own\ncross-validation scheme, so :func:`~skore.evaluate` needs no ``splitter``.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data = skrub.var(\"data\", df_batch)\ngroups = data[\"batch_id\"]\nX_op = data.drop(columns=[\"batch_id\", \"target\"]).skb.mark_as_X(\n cv=GroupKFold(n_splits=5),\n split_kwargs={\"groups\": groups},\n)\ny_op = data[\"target\"].skb.mark_as_y()\nlearner = X_op.skb.apply(model, y=y_op).skb.make_learner()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_grouped = skore.evaluate(learner, data={\"data\": df_batch}, pos_label=1)\nreport_grouped.metrics.summarize(data_source=\"test\").frame(\n aggregate=None, flat_index=False\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking at the checks results, ``SKD003`` correctly flags split #0.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_grouped.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Distribution shift in the last time-series fold\n\nUnder :class:`~sklearn.model_selection.TimeSeriesSplit`, later windows can diverge\nfrom the training data. Consider a medical setting where diagnoses are collected over\ntime: a sudden influx of ill patients near the end of the study shifts the class\ndistribution. Earlier folds look strong because the model was trained on a balanced\npopulation, but the last fold drops as the positive class becomes rare. SKD003 fires,\nwhich is expected from chronological cross-validation and not a reason to reshuffle\ntime.\n\nWe reuse the same breast-cancer dataset, attach a fake timestamp, and reduce the\nprevalence of the positive class in the last test window.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.model_selection import TimeSeriesSplit\n\nn_splits_time = 5\nlast_test_start = len(X) - (len(X) // (n_splits_time + 1))\n\ny_time = y.copy()\nrng_time = np.random.default_rng(seed=2)\ny_time.iloc[last_test_start:] = rng_time.choice(\n [0, 1], size=len(X) - last_test_start, p=[0.95, 0.05]\n)\n\ntimestamps = pd.date_range(\"2020-01-01\", periods=len(X), freq=\"D\")\ndf_time = X.assign(timestamp=timestamps, target=y_time)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As for the grouped section, we use a :class:`~skrub.DataOp` to declare the\ntime-series split directly on the data.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data_time = skrub.var(\"data_time\", df_time)\nX_time_op = data_time.drop(columns=[\"timestamp\", \"target\"]).skb.mark_as_X(\n cv=TimeSeriesSplit(n_splits=n_splits_time),\n)\ny_time_op = data_time[\"target\"].skb.mark_as_y()\nlearner_time = X_time_op.skb.apply(model, y=y_time_op).skb.make_learner()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_time = skore.evaluate(learner_time, data={\"data_time\": df_time}, pos_label=1)\nreport_time.metrics.summarize(data_source=\"test\").frame(\n aggregate=None, flat_index=False\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Split #4 underperforms because the positive class is now rare in that window.\nSKD003 here is expected from chronological cross-validation, not a reason to\nreshuffle time.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_time.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# When the problem is understood: mute SKD003\n\nGroups and drift are data properties: investigate, collect more labels on the\nhard regime if needed, but folds may never look uniform. Once SKD003 is\nexpected, mute it with :func:`~skore.configuration` (or\n``ignore=[\"SKD003\"]`` on one summarize call).\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "with skore.configuration(ignore_checks=[\"SKD003\"]):\n muted = report_time.checks.summarize(fast_mode=True)\nmuted" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Side note: if there is no group or time structure, but an unshuffled\n:class:`~sklearn.model_selection.KFold` still creates imbalanced folds because\nclass prevalence varies along the collection order (for example a sensor\nfailed for part of the dump), shuffling or stratifying is appropriate *when\nyou are sure that irregularity is accidental and will not appear in\nproduction*. That case is the exception where changing the splitter to\nsmooth folds is the right fix; do not use it to hide real groups or time\ndrift.\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n\nSKD003 is a reminder to inspect unstable cross-validation folds. With a proper\nsplitter, firing often means the evaluation exposed a bad batch, a hard group, or\ntemporal shift. Fix what you can (for example a corrupted label window). When the\nstructure is intrinsic, keep the honest splitter, document the outlier regime, mute\nSKD003 via configuration, and collect more data on that regime if you need better\ncoverage. Avoid shuffled cross-validation as a way to make the check disappear when\ngroups or time are real. See also `SKD013 ` for\nchronological train/test overlap on hold-out reports.\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.7" } }, "nbformat": 4, "nbformat_minor": 0 }