SKD007 - MDI feature importance is biased for high-cardinality features

This example demonstrates the limitations that SKD007 warns against on tree models. Mean decrease in impurity (MDI) tends to rank high-cardinality categorical or continuous columns as more important than features with as much signal but lower cardinality. This is due to the tree building process picking high cardinality features more often as they offer more split points to choose from.

Mitigations from the Automated checks user guide:

• use permutation importance instead of MDI,

• cross-check MDI with permutation importance or drop-column importance.

We will compare MDI to permutation importance to show that it gives a more reliable estimate of feature importance. The same contrast is illustrated in scikit-learn’s Permutation Importance vs Random Forest Feature Importance (MDI) example.

We fit a RandomForestRegressor on a 1,500-row subsample of California housing. The goal is to show the limitations of impurity based importance and show they do not affect permutation importance on a test set.

Load the California housing dataset

Continuous columns such as AveRooms and AveOccup take many distinct values: above the 50 % of samples threshold SKD007 uses for high-cardinality features.

import numpy as np
from sklearn.model_selection import train_test_split
from skrub.datasets import fetch_california_housing

housing = fetch_california_housing()
X_full, y_full = housing.X, housing.y

X, _, y, _ = train_test_split(
    X_full,
    y_full,
    train_size=1_500,
    random_state=42,
)

Let us add two random features that carry no signal about the target: a continuous draw from a normal distribution, and a categorical feature with 20 levels sampled uniformly (stored as integer codes so the forest can split on them directly). High-cardinality noise can still receive non-zero MDI, and often more MDI than low-cardinality noise, while permutation importance on the test set should stay near zero for both.

rng = np.random.default_rng(42)
X["noise_cont"] = rng.normal(size=len(X))
X["noise_cat"] = rng.integers(0, 20, size=len(X))

Let us inspect the feature matrix with TableReport.

from skrub import TableReport

TableReport(X)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

TableReport(y)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Counting unique values per column previews which features SKD007 will flag.

X.nunique().sort_values(ascending=False)

noise_cont    1500
AveRooms      1491
AveOccup      1478
AveBedrms     1408
MedInc        1380
Population    1128
Longitude      483
Latitude       458
HouseAge        51
noise_cat       20
dtype: int64

Trigger SKD007 with a random forest on continuous features

A random forest exposes feature_importances_ based on MDI. After fitting, let us inspect impurity decrease with skore.

from sklearn.ensemble import RandomForestRegressor
from skore import TrainTestSplit, evaluate

splitter = TrainTestSplit(random_state=42)

report = evaluate(
    RandomForestRegressor(random_state=42),
    X=X,
    y=y,
    splitter=splitter,
)
report

RandomForestRegressor(random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

SKD007 warns about high-cardinality columns such as MedInc and AveOccup. The synthetic continuous noise column is high-cardinality as well, so it belongs in the same tip.

report.checks.summarize(fast_mode=True)

Let us plot MDI with features sorted by importance.

import matplotlib.pyplot as plt

mdi_display = report.inspection.impurity_decrease()
_ = mdi_display.plot(sorting_order="descending")

The two synthetic noise columns are not near zero under MDI: impurity still assigns them mass. noise_cont in particular is high-cardinality, so the forest can keep finding splits on it even though it carries no target signal.

Use permutation importance instead of MDI

permutation_importance() shuffles each column on the test set and measures the score drop. The result is not biased toward high-cardinality split points the way MDI is.

perm_display = report.inspection.permutation_importance(
    seed=42,
    n_repeats=5,
)
_ = perm_display.plot(sorting_order="descending")

Under permutation importance the noisy features sit at (or very near) zero: shuffling them does not change the test score, so they are not contributing to predicting the target.

Cross-check MDI with permutation importance

Let us put the two rankings side by side. Sorting by MDI tends to push high-cardinality columns (including noise_cont) upward; permutation importance on the test set should keep both synthetic features near the bottom even when MDI does not.

mdi = (
    mdi_display.frame(sorting_order="descending")
    .set_index("feature")
    .rename(columns={"importance": "mdi"})
)
perm = (
    perm_display.frame(sorting_order="descending")
    .set_index("feature")[["value_mean"]]
    .rename(columns={"value_mean": "permutation"})
)
nunique = X.nunique().rename("nunique")

comparison = (
    mdi.join(perm)
    .join(nunique)
    .assign(
        mdi_rank=lambda df: df["mdi"].rank(ascending=False).astype(int),
        perm_rank=lambda df: df["permutation"].rank(ascending=False).astype(int),
    )
    .sort_values("mdi", ascending=False)
)
comparison

	mdi	permutation	nunique	mdi_rank	perm_rank
feature
MedInc	0.522390	1.134807	1380	1	1
AveOccup	0.140019	0.128296	1478	2	2
HouseAge	0.067881	0.050188	51	3	5
Longitude	0.054675	0.082798	483	4	4
Latitude	0.050791	0.128265	458	5	3
AveRooms	0.049150	0.044559	1491	6	6
AveBedrms	0.037725	0.014047	1408	7	7
Population	0.031908	0.011334	1128	8	8
noise_cont	0.028676	0.004603	1500	9	9
noise_cat	0.016786	0.001149	20	10	10

The side-by-side bars make the disagreement easier to read: impurity can assign mass to noise_cont (and sometimes more than to noise_cat), while permutation importance stays close to zero for both.

fig, axes = plt.subplots(1, 2, figsize=(12, 5), sharey=True)
order = comparison.sort_values("mdi", ascending=True).index

axes[0].barh(order, comparison.loc[order, "mdi"])
axes[0].set_title("MDI (impurity decrease)")
axes[0].set_xlabel("Importance")

axes[1].barh(order, comparison.loc[order, "permutation"])
axes[1].set_title("Permutation importance (test)")
axes[1].set_xlabel("Mean score drop")

fig.tight_layout()
_ = fig

Conclusion

SKD007 warns that MDI feature importance favors high-cardinality inputs such as AveOccup and can inflate the role of irrelevant high-cardinality noise. In this walkthrough, permutation importance gave a more reliable picture of which features actually move test scores. When importance is a decision factor, we prefer permutation (or drop-column tests) over impurity alone.