{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# SKD004 - High class imbalance\n\n`SKD004 <skd004-high-class-imbalance>` flags a binary classification task\nwhen the majority class exceeds 80 % of rows. Accuracy can look high while\nthe minority class is ignored as a default. This notebook is\nmostly about how to work with that imbalance once the check fires: we do not\ntry to make SKD004 disappear, because natural prevalence is often the right\nthing to keep.\n\nWhat we do instead (see also `automated_checks`):\n\n- report absolute counts as well as percentages,\n- evaluate ranking and calibration (ROC AUC, log-loss) before trusting\n  thresholded precision / recall,\n- tune the decision threshold under an explicit precision / recall or cost\n  constraint (for example with\n  :class:`~sklearn.model_selection.TunedThresholdClassifierCV`),\n- avoid ``class_weight`` and resampling when calibrated probabilities matter,\n- correct for prevalence shift if you collect minority-only data.\n\nSee also:\nhttps://probabl-ai.github.io/calibration-cost-sensitive-learning/content/notebooks/imbalanced_classification.html\n\nWe use Covertype forest types 2 (majority) vs 5 (minority) on an 8,000-row\nstratified subsample. The goal is to keep natural prevalence, judge probability\nquality first, then choose a cut-off that matches the precision / recall\ntrade-off you care about.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Load Covertype (types 2 vs 5)\n\nTypes 2 vs 5 give a natural imbalance (type 2 is the majority class). We keep\nminority type 5 as the positive class and draw an 8,000-row stratified\nsubsample so the gallery stays fast while absolute minority counts remain\nlarge enough to learn from.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nfrom sklearn.datasets import fetch_covtype\nfrom sklearn.model_selection import train_test_split\n\ndf = fetch_covtype(as_frame=True).frame\npair = df.query(\"Cover_Type.isin([2, 5])\")\ny_full = (pair[\"Cover_Type\"] == 5).astype(int).rename(\"is_type_5\")\nX_full = pair.drop(columns=[\"Cover_Type\"])\n\nX, _, y, _ = train_test_split(\n    X_full,\n    y_full,\n    train_size=8_000,\n    stratify=y_full,\n    random_state=42,\n)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "y.value_counts(normalize=True).round(4)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "y.value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Inspect the feature matrix with :class:`~skrub.TableReport`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport\n\nTableReport(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The binary target marks type-5 stands; the majority class exceeds 80 % of\nrows, so SKD004 will fire.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "TableReport(y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Trigger SKD004 - default classifier on imbalanced labels\n\nA default gradient boosting classifier does not change label counts. The\ncheck cares about the class mix in the data, not about whether we reweighted\nthe loss.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import HistGradientBoostingClassifier\nfrom skore import TrainTestSplit, evaluate\n\nsplitter = TrainTestSplit(test_size=0.2, random_state=42, stratify=y)\nclassifier = HistGradientBoostingClassifier(random_state=42)\n\nreport = evaluate(\n    classifier,\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter,\n)\nreport"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD004 should fire: the majority class exceeds 80 % of rows. Ignore SKD008 to\navoid correlated-feature warnings from Covertype's constant soil one-hots.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.checks.summarize(fast_mode=True, ignore=[\"SKD008\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Accuracy and F1 are poor defaults under imbalance\n\nAccuracy can be inflated by nearly always predicting the majority class. F1\naverages precision and recall into one number and hides which side of the\ntrade-off you care about, so we do not use it here. At the default\nprobability cut-off of 0.5, minority recall is often weak because rare\nevents receive small predicted probabilities.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize(\n    metric=[\"accuracy\", \"precision\", \"recall\", \"roc_auc\", \"log_loss\"],\n    data_source=\"both\",\n).frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The stratified hold-out still has only a few dozen type-5 rows against about\n1,500 majority rows. Most of those scarce positives are predicted as\nmajority, so minority recall is low while accuracy stays high.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.confusion_matrix().plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Check ranking and calibration first\n\nBefore touching thresholds, ask whether probabilities are any good:\n\n- ROC AUC asks whether positives tend to get higher scores than negatives\n  (threshold-free ranking),\n- log-loss penalizes confident wrong probabilities,\n- a calibration curve asks whether predicted probabilities match observed\n  frequencies.\n\nOn the calibration plot, bins of predicted probability are compared to the\nfraction of true positives in each bin. A useful curve hugs the diagonal: when\nthe model says \"20 %\", about 20 % of those rows really are positive. Points\nabove the diagonal mean under-confidence (events happen more often than\npredicted); points below mean over-confidence (the model is too sure). With a\nrare class, almost all mass sits at low probabilities, so the curve often\nonly appears on the left of the plot; that is expected, not a plotting bug.\n\nIf ranking and calibration look reasonable, the model may already be useful;\nthe default 0.5 cut-off is simply the wrong operating point for a rare class.\nThe next section shows how ``class_weight=\"balanced\"`` can push the curve\nbelow the diagonal by inflating minority probabilities.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize(\n    metric=[\"roc_auc\", \"log_loss\"],\n    data_source=\"test\",\n).frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.inspection.calibration_curve(data_source=\"test\", n_bins=10).plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Class weights as a cautionary comparison\n\nA common reflex is ``class_weight=\"balanced\"``. Rebalancing with weights is\nequivalent in spirit to resampling methods such as SMOTE or random\noversampling / undersampling: they change the effective class mix and will\nsuffer from the same issues. That often improves precision / recall at 0.5\nbecause it inflates minority probabilities, but it typically breaks\ncalibration: predicted probabilities run ahead of observed rates, so the\ncurve drifts below the diagonal (over-confidence on the originally rare\nclass). If you later recalibrate, the thresholded gains often disappear. We\nshow the comparison, then leave weights behind when calibrated probabilities\nmatter.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import compare\n\nreport_weighted = evaluate(\n    HistGradientBoostingClassifier(class_weight=\"balanced\", random_state=42),\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter,\n)\n\ncomparison_weights = compare(\n    {\"default\": report, \"class_weight_balanced\": report_weighted}\n)\ncomparison_weights.metrics.summarize(\n    metric=[\"precision\", \"recall\", \"roc_auc\", \"log_loss\"],\n    data_source=\"test\",\n).frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After reweighting, compare this curve to the default one: points tend to sit\nfurther below the diagonal (over-confident on the rare class).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_weighted.inspection.calibration_curve(data_source=\"test\", n_bins=10).plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Tune the decision threshold\n\nKeep the default, prevalence-correct model and change only the decision rule.\nFor this demo we require at least 30 % precision on type 5, then maximize\nrecall. That floor is an explicit product choice: high enough to limit false\nalarms, low enough that a rare-event model can still catch a useful share of\ntrue type-5 stands. Replace 0.3 with a cost or capacity constraint in real\nwork.\n\n:class:`~sklearn.model_selection.TunedThresholdClassifierCV` searches the\ncut-off by cross-validation and does not change ``predict_proba``, so\ncalibration stays intact.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import make_scorer, precision_score, recall_score\nfrom sklearn.model_selection import TunedThresholdClassifierCV\n\n\ndef recall_with_min_precision(y_true, y_pred, precision_level=0.3):\n    \"\"\"Maximize recall only among thresholds that keep precision high enough.\"\"\"\n    precision = precision_score(y_true, y_pred, zero_division=0)\n    recall = recall_score(y_true, y_pred, zero_division=0)\n    if precision < precision_level:\n        return -np.inf\n    return recall\n\n\nthreshold_scoring = make_scorer(recall_with_min_precision, precision_level=0.3)\n\ntuned = TunedThresholdClassifierCV(\n    estimator=HistGradientBoostingClassifier(random_state=42),\n    scoring=threshold_scoring,\n    cv=3,\n    n_jobs=4,\n)\n\nreport_tuned = evaluate(\n    tuned,\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD004 still fires: label counts did not change. That is expected. We\nimproved how we decide, not the histogram SKD004 reads. Ignore SKD008 to\navoid correlated-feature warnings from Covertype's constant soil one-hots.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tuned.checks.summarize(fast_mode=True, ignore=[\"SKD008\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(\"Chosen decision threshold:\", float(report_tuned.estimator_.best_threshold_))\n\nreport_tuned.metrics.summarize(\n    metric=[\"accuracy\", \"precision\", \"recall\", \"roc_auc\", \"log_loss\"],\n    data_source=\"both\",\n).frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tuned.metrics.confusion_matrix().plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Compare the default 0.5 cut-off to the tuned threshold. Same underlying\nprobabilities; only the hard predictions change. Precision / recall move;\nROC AUC and log-loss stay essentially the same.\n\nThe scorer asks for precision of at least 0.3, then maximizes recall: catch\nmore type-5 stands without too many false alarms (fraud review, maintenance\ntickets, medical triage). Preferring high precision instead fits cases where\na false alarm is costly: auto-blocking users, expensive tests, or limited\noutreach budgets.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "comparison_thresholds = compare(\n    {\n        \"default_threshold_0.5\": report,\n        \"tuned_threshold\": report_tuned,\n    }\n)\ncomparison_thresholds.metrics.summarize(\n    metric=[\"precision\", \"recall\", \"roc_auc\", \"log_loss\"],\n    data_source=\"test\",\n).frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Inspect the precision-recall curve\n\nThe dashed line is our precision floor (0.3). The tuned threshold should land\nnear the highest-recall point that still respects that floor.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "threshold = float(report_tuned.estimator_.best_threshold_)\n\ndisplay = report.metrics.precision_recall()\nfig = display.plot()\nax = fig.axes[0]\nax.axhline(0.3, linestyle=\"--\", color=\"gray\", label=\"precision floor 0.3\")\nax.axvline(\n    report_tuned.metrics.recall(),\n    linestyle=\":\",\n    color=\"C1\",\n    label=f\"recall at threshold={threshold:.3f}\",\n)\nax.legend(loc=\"best\")\nax.set_title(\"Precision-recall curve (test fold)\")\nfig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Collecting more minority data\n\nGathering more type-5 plots can help the model see the rare class. If\nacquisition preferentially samples minority rows, train prevalence no longer\nmatches production. Probabilities and thresholds fitted on that mix will be\nbiased unless you correct for the shift. Clearing SKD004 by stuffing\nminority rows into the table is therefore not automatically a success.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Conclusion\n\nSKD004 warns that one class dominates the table. Keep natural prevalence when\nyou need honest probabilities; move the threshold when you need a different\nprecision / recall trade-off. Class weights and resampling are risky\nshortcuts if calibration matters for the decisions you deploy.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}