{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# SKD007 - MDI feature importance is biased for high-cardinality features\n\nThis example demonstrates the limitations that\n`SKD007 <skd007-mdi-cardinality-bias>` warns against on tree models. Mean\ndecrease in impurity (MDI) tends to rank high-cardinality categorical or\ncontinuous columns as more important than features with as much signal but\nlower cardinality. This is due to the tree building process picking high\ncardinality features more often as they offer more split points to choose from.\n\nMitigations from the `automated_checks` user guide:\n\n- use permutation importance instead of MDI,\n- cross-check MDI with permutation importance or drop-column importance.\n\nWe will compare MDI to permutation importance to show that it gives a more\nreliable estimate of feature importance. The same contrast is illustrated in\nscikit-learn's\n[Permutation Importance vs Random Forest Feature Importance (MDI)](https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance.html)\nexample.\n\nWe fit a :class:`~sklearn.ensemble.RandomForestRegressor` on a 1,500-row\nsubsample of California housing. The goal is to show the limitations of\nimpurity based importance and show they do not affect permutation importance\non a test set.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Load the California housing dataset\n\nContinuous columns such as ``AveRooms`` and ``AveOccup`` take many distinct\nvalues: above the 50 % of samples threshold SKD007 uses for high-cardinality\nfeatures.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom skrub.datasets import fetch_california_housing\n\nhousing = fetch_california_housing()\nX_full, y_full = housing.X, housing.y\n\nX, _, y, _ = train_test_split(\n    X_full,\n    y_full,\n    train_size=1_500,\n    random_state=42,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let us add two random features that carry no signal about the target: a\ncontinuous draw from a normal distribution, and a categorical feature with 20\nlevels sampled uniformly (stored as integer codes so the forest can split on\nthem directly). High-cardinality noise can still receive non-zero MDI, and\noften more MDI than low-cardinality noise, while permutation importance on the\ntest set should stay near zero for both.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "rng = np.random.default_rng(42)\nX[\"noise_cont\"] = rng.normal(size=len(X))\nX[\"noise_cat\"] = rng.integers(0, 20, size=len(X))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let us inspect the feature matrix with :class:`~skrub.TableReport`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport\n\nTableReport(X)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "TableReport(y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Counting unique values per column previews which features SKD007 will flag.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "X.nunique().sort_values(ascending=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Trigger SKD007 with a random forest on continuous features\n\nA random forest exposes ``feature_importances_`` based on MDI. After fitting,\nlet us inspect impurity decrease with skore.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestRegressor\nfrom skore import TrainTestSplit, evaluate\n\nsplitter = TrainTestSplit(random_state=42)\n\nreport = evaluate(\n    RandomForestRegressor(random_state=42),\n    X=X,\n    y=y,\n    splitter=splitter,\n)\nreport"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD007 warns about high-cardinality columns such as ``MedInc`` and\n``AveOccup``. The synthetic continuous noise column is high-cardinality as\nwell, so it belongs in the same tip.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let us plot MDI with features sorted by importance.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n\nmdi_display = report.inspection.impurity_decrease()\n_ = mdi_display.plot(sorting_order=\"descending\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The two synthetic noise columns are not near zero under MDI: impurity still\nassigns them mass. ``noise_cont`` in particular is high-cardinality, so the\nforest can keep finding splits on it even though it carries no target signal.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Use permutation importance instead of MDI\n\n:meth:`~skore.EstimatorReport.inspection.permutation_importance` shuffles each\ncolumn on the test set and measures the score drop. The result is not biased\ntoward high-cardinality split points the way MDI is.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "perm_display = report.inspection.permutation_importance(\n    seed=42,\n    n_repeats=5,\n)\n_ = perm_display.plot(sorting_order=\"descending\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Under permutation importance the noisy features sit at (or very near) zero:\nshuffling them does not change the test score, so they are not contributing\nto predicting the target.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Cross-check MDI with permutation importance\n\nLet us put the two rankings side by side. Sorting by MDI tends to push\nhigh-cardinality columns (including ``noise_cont``) upward; permutation\nimportance on the test set should keep both synthetic features near the\nbottom even when MDI does not.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "mdi = (\n    mdi_display.frame(sorting_order=\"descending\")\n    .set_index(\"feature\")\n    .rename(columns={\"importance\": \"mdi\"})\n)\nperm = (\n    perm_display.frame(sorting_order=\"descending\")\n    .set_index(\"feature\")[[\"value_mean\"]]\n    .rename(columns={\"value_mean\": \"permutation\"})\n)\nnunique = X.nunique().rename(\"nunique\")\n\ncomparison = (\n    mdi.join(perm)\n    .join(nunique)\n    .assign(\n        mdi_rank=lambda df: df[\"mdi\"].rank(ascending=False).astype(int),\n        perm_rank=lambda df: df[\"permutation\"].rank(ascending=False).astype(int),\n    )\n    .sort_values(\"mdi\", ascending=False)\n)\ncomparison"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The side-by-side bars make the disagreement easier to read: impurity can\nassign mass to ``noise_cont`` (and sometimes more than to ``noise_cat``),\nwhile permutation importance stays close to zero for both.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig, axes = plt.subplots(1, 2, figsize=(12, 5), sharey=True)\norder = comparison.sort_values(\"mdi\", ascending=True).index\n\naxes[0].barh(order, comparison.loc[order, \"mdi\"])\naxes[0].set_title(\"MDI (impurity decrease)\")\naxes[0].set_xlabel(\"Importance\")\n\naxes[1].barh(order, comparison.loc[order, \"permutation\"])\naxes[1].set_title(\"Permutation importance (test)\")\naxes[1].set_xlabel(\"Mean score drop\")\n\nfig.tight_layout()\n_ = fig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Conclusion\n\nSKD007 warns that MDI feature importance favors high-cardinality inputs such\nas ``AveOccup`` and can inflate the role of irrelevant high-cardinality noise.\nIn this walkthrough, permutation importance gave a more reliable picture of\nwhich features actually move test scores. When importance is a decision\nfactor, we prefer permutation (or drop-column tests) over impurity alone.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}