{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n\n# SKD005 - Underrepresented classes\n\n`SKD005 ` flags a multiclass task when\none or more classes each represent less than 10 % of rows. Overall accuracy can\nlook acceptable while rare labels are barely learned. This notebook is about\nhow to work with that rarity once the check fires: we do not try to make SKD005\ndisappear by reshaping the class histogram.\n\nWhat to do instead:\n\n- report absolute counts as well as percentages,\n- evaluate threshold-free / probabilistic metrics (such as log-loss) before\n per class precision and accuracy,\n- collect more rare-class labels when possible, without treating a cleared\n check as success,\n- correct for prevalence shift if acquisition oversamples rare types.\n\nFor binary rare-event tasks (threshold tuning, when ``class_weight`` is a risky\nshortcut), see `skd004-high-class-imbalance` and:\nhttps://probabl-ai.github.io/calibration-cost-sensitive-learning/content/notebooks/imbalanced_classification.html\n\nWe take a 10,000-row stratified subsample of Covertype so several forest types\nfall below 10 %. The goal is to keep natural prevalence visible, judge the\nmulticlass model honestly, then see what extra rare-class rows can do.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load the Covertype dataset\n\nThe full Covertype task has seven forest types. A small stratified subsample\nkeeps frequent classes well represented while types 3-7 drop below 10 % each.\nWe keep the unused rows as a pool for the \"more rare-class data\" section later.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import pandas as pd\nfrom sklearn.datasets import fetch_covtype\nfrom sklearn.model_selection import train_test_split\n\ndf = fetch_covtype(as_frame=True).frame\ny_full = df[\"Cover_Type\"].astype(str)\nX_full = df.drop(columns=[\"Cover_Type\"])\n\nX, X_pool, y, y_pool = train_test_split(\n X_full,\n y_full,\n train_size=10_000,\n stratify=y_full,\n random_state=42,\n)\ny, y_pool = y.rename(\"class\"), y_pool.rename(\"class\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us inspect the feature matrix with :class:`~skrub.TableReport`.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from skrub import TableReport\n\nTableReport(X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Clicking the column in the target's `TableReport` brings\na class histogram that shows that the classes are not evenly distributed.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "TableReport(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us also look at absolute counts: type 4 is under 1 % with only a\nfew dozen rows in this subsample: even a good multiclass model has little to\nlearn from there. Types such as 6 are also under the 10 % SKD005 bar, but with\na few hundred rows they are less hopeless.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "shares = y.value_counts(normalize=True).sort_index()\ncounts = y.value_counts().sort_index()\npd.concat([shares.round(4), counts], axis=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Trigger SKD005: default classifier on imbalanced classes\n\nA default gradient boosting classifier does not change label counts. We ignore\nSKD008 to avoid correlated-feature warnings from Covertype's constant soil\none-hots.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import skore\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom skore import TrainTestSplit\n\nsplitter = TrainTestSplit(test_size=0.2, random_state=42, stratify=y)\nclassifier = HistGradientBoostingClassifier(random_state=42)\n\nreport = skore.evaluate(\n classifier,\n X=X,\n y=y,\n splitter=splitter,\n)\nreport" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SKD005 correctly flags classes 3, 4, 5, 6, and 7 as under 10 % of rows.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.checks.summarize(fast_mode=True, ignore=[\"SKD008\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Accuracy hides rare-class failures\n\nAccuracy alone can look strong when frequent classes dominate the table.\nLet us however report it with per class precision and log-loss.\nWe see that global accuracy hides rare-class failures, such as for types\n4 and 5.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.metrics.summarize(metric=[\"accuracy\", \"precision\", \"log_loss\"]).frame(\n flat_index=False\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us also inspect the confusion matrix. We see that the model has a hard time\npredicting type 6, often confusing it with types 2 and 3 or when predicting type 7,\nconfusing it with type 1.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.metrics.confusion_matrix().plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# More rare-class training data\n\nExtra labels on underrepresented types can help the multiclass model see them\nmore often. Let us keep one fixed test fold with the natural mix, fit on the\noriginal train fold, then refit after adding rare-class rows from the pool\n(classes that were under 10 % in the subsample).\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "X_train, X_test, y_train, y_test = train_test_split(\n X, y, test_size=0.2, random_state=42, stratify=y\n)\n\nrare_labels = shares[shares < 0.10].index\npool_rare = y_pool.isin(rare_labels)\nX_rare_extra, _, y_rare_extra, _ = train_test_split(\n X_pool.loc[pool_rare],\n y_pool.loc[pool_rare],\n train_size=min(5_000, int(pool_rare.sum())),\n stratify=y_pool.loc[pool_rare],\n random_state=42,\n)\nX_train_more = pd.concat([X_train, X_rare_extra])\ny_train_more = pd.concat([y_train, y_rare_extra])\n\nprint(\"Rare labels added from the pool:\", list(rare_labels))\nprint(\"Extra rare-class rows added:\", len(y_rare_extra))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "print(\"\\nBaseline train counts:\")\ny_train.value_counts().sort_index()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "print(\"\\nEnriched train counts:\")\ny_train_more.value_counts().sort_index()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us now fit a model on the enriched train set and compare the results with the\noriginal model. We can observe that the model on the enriched train set has a better\nlog-loss, accuracy and per-class precision on the common test set.\n\nThe log-loss is the most importance metric to look at here, as it evaluates\nthe model's predicted probabilities, which give more robust estimate of the\nmodel's quality. In contrast, accuracy and per-class precision are computed\nwith hard class predictions, obtained from the argmax of the predicted\nprobabilities, which can hide uncalibrated predictions.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model_less = HistGradientBoostingClassifier(random_state=42).fit(X_train, y_train)\nreport_less = skore.evaluate(model_less, X_test, y_test, splitter=\"prefit\")\nreport_less" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model_more = HistGradientBoostingClassifier(random_state=42).fit(\n X_train_more, y_train_more\n)\nreport_more = skore.evaluate(model_more, X_test, y_test, splitter=\"prefit\")\nreport_more" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "comparison_report = skore.compare(\n {\n \"baseline_train\": report_less,\n \"more_rare_class_rows\": report_more,\n }\n)\ncomparison_report.metrics.summarize(\n metric=[\"accuracy\", \"precision\", \"log_loss\"],\n data_source=\"test\",\n).frame(flat_index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Enriching the train set with more rare-class rows also clears SKD005.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_more.checks.summarize(fast_mode=True, ignore=[\"SKD008\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Collecting more rare-class rows can improve rare-class precision and log-loss on\na fixed natural-prevalence test set. That does not mean we should chase a\ncleared SKD005: if acquisition preferentially samples rare types, the training\nmix no longer matches the field, and production prevalence may stay low. We\ncan correct for that shift before reading operating metrics. Clearing the\ncheck by reshaping the histogram is optional; better rare-class decisions\nunder honest prevalence is the point.\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n\nSKD005 is a multiclass rarity warning, not a request to rebalance at all\ncosts. We prefer log-loss (and confusion matrices) over accuracy, we report\nabsolute counts, and we can add rare-class labels when we can without treating\na silent check as success.\n\nFor binary rare-event threshold tuning and when\n``class_weight`` is a risky shortcut, see\n`skd004-high-class-imbalance`.\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.7" } }, "nbformat": 4, "nbformat_minor": 0 }