.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/pitfalls_and_solutions/plot_skd007_mdi_cardinality_bias.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_pitfalls_and_solutions_plot_skd007_mdi_cardinality_bias.py: .. _example_skd007_mdi_cardinality_bias: SKD007 - MDI feature importance is biased for high-cardinality features ======================================================================= This example demonstrates the limitations that :ref:`SKD007 ` warns against on tree models. Mean decrease in impurity (MDI) tends to rank high-cardinality categorical or continuous columns as more important than features with as much signal but lower cardinality. This is due to the tree building process picking high cardinality features more often as they offer more split points to choose from. Mitigations from the :ref:`automated_checks` user guide: - use permutation importance instead of MDI, - cross-check MDI with permutation importance or drop-column importance. We will compare MDI to permutation importance to show that it gives a more reliable estimate of feature importance. The same contrast is illustrated in scikit-learn's `Permutation Importance vs Random Forest Feature Importance (MDI) `_ example. We fit a :class:`~sklearn.ensemble.RandomForestRegressor` on a 1,500-row subsample of California housing. The goal is to show the limitations of impurity based importance and show they do not affect permutation importance on a test set. .. GENERATED FROM PYTHON SOURCE LINES 33-39 Load the California housing dataset =================================== Continuous columns such as ``AveRooms`` and ``AveOccup`` take many distinct values: above the 50 % of samples threshold SKD007 uses for high-cardinality features. .. GENERATED FROM PYTHON SOURCE LINES 39-54 .. code-block:: Python import numpy as np from sklearn.model_selection import train_test_split from skrub.datasets import fetch_california_housing housing = fetch_california_housing() X_full, y_full = housing.X, housing.y X, _, y, _ = train_test_split( X_full, y_full, train_size=1_500, random_state=42, ) .. GENERATED FROM PYTHON SOURCE LINES 55-61 Let us add two random features that carry no signal about the target: a continuous draw from a normal distribution, and a categorical feature with 20 levels sampled uniformly (stored as integer codes so the forest can split on them directly). High-cardinality noise can still receive non-zero MDI, and often more MDI than low-cardinality noise, while permutation importance on the test set should stay near zero for both. .. GENERATED FROM PYTHON SOURCE LINES 61-66 .. code-block:: Python rng = np.random.default_rng(42) X["noise_cont"] = rng.normal(size=len(X)) X["noise_cat"] = rng.integers(0, 20, size=len(X)) .. GENERATED FROM PYTHON SOURCE LINES 67-68 Let us inspect the feature matrix with :class:`~skrub.TableReport`. .. GENERATED FROM PYTHON SOURCE LINES 68-73 .. code-block:: Python from skrub import TableReport TableReport(X) .. raw:: html

	MedInc	HouseAge	AveRooms	AveBedrms	Population	AveOccup	Latitude	Longitude	noise_cont	noise_cat
1,763	3.67	45.0	4.18	0.889	702.	2.60	38.0	-122.	0.305	8
18,259	5.08	36.0	6.03	0.995	485.	2.44	37.4	-122.	-1.04	16
13,547	1.17	37.0	4.02	0.977	1.13e+03	3.21	34.1	-117.	0.750	5
11,536	1.72	27.0	3.67	1.02	847.	1.35	33.8	-118.	0.941	4
15,069	2.98	35.0	4.52	0.959	681.	2.77	32.8	-117.	-1.95	6

11,284	6.37	35.0	6.13	0.926	658.	3.03	33.8	-118.	-1.10	15
11,964	3.05	33.0	6.87	1.27	1.75e+03	3.90	34.0	-117.	0.323	1
5,390	2.93	36.0	3.99	1.08	1.76e+03	3.33	34.0	-118.	-1.15	3
860	5.72	15.0	6.40	1.07	1.78e+03	3.18	37.6	-122.	0.712	5
15,795	2.58	52.0	3.40	1.06	2.62e+03	2.11	37.8	-122.	-1.18	5

Column	Column name	dtype	Is sorted	Unique values	Mean	Std	Min	Median	Max
0	MedInc	Float64DType	False	1380 (92.0%)	3.86	1.89	0.536	3.51	15.0
1	HouseAge	Float64DType	False	51 (3.4%)	28.3	12.5	2.00	28.0	52.0
2	AveRooms	Float64DType	False	1491 (99.4%)	5.43	2.17	1.63	5.26	47.5
3	AveBedrms	Float64DType	False	1408 (93.9%)	1.10	0.421	0.810	1.05	11.2
4	Population	Float64DType	False	1128 (75.2%)	1.43e+03	1.07e+03	3.00	1.17e+03	1.22e+04
5	AveOccup	Float64DType	False	1478 (98.5%)	2.94	0.921	0.750	2.83	21.3
6	Latitude	Float64DType	False	458 (30.5%)	35.5	2.09	32.6	34.2	41.8
7	Longitude	Float64DType	False	483 (32.2%)	-119.	1.97	-124.	-118.	-114.
8	noise_cont	Float64DType	False	1500 (100.0%)	-0.0206	1.00	-3.65	0.00380	3.18
9	noise_cat	Int64DType	False	20 (1.3%)	9.39	5.76	0	9	19

Column 1	Column 2	Cramér's V	Pearson's Correlation
AveRooms	AveBedrms	0.720	0.809
Latitude	Longitude	0.535	-0.924
MedInc	AveRooms	0.253	0.395
HouseAge	Longitude	0.176	-0.0476
AveRooms	Longitude	0.160	-0.0384
AveOccup	noise_cont	0.145	0.0581
HouseAge	Latitude	0.136	-0.0398
HouseAge	Population	0.135	-0.292
AveBedrms	Longitude	0.131	0.0324
AveOccup	Longitude	0.121	0.157
Population	AveOccup	0.121	0.176
AveOccup	Latitude	0.116	-0.147
HouseAge	AveRooms	0.106	-0.187
MedInc	HouseAge	0.102	-0.147
AveRooms	Latitude	0.101	0.124
AveOccup	noise_cat	0.0977	-0.0309
MedInc	Latitude	0.0967	-0.0541
AveBedrms	Latitude	0.0954	0.0609
HouseAge	AveOccup	0.0922	0.0372
AveBedrms	noise_cat	0.0889	-0.0226
Longitude	noise_cat	0.0868	0.0206
Latitude	noise_cat	0.0864	-0.0164
AveRooms	noise_cat	0.0862	-0.00400
noise_cont	noise_cat	0.0859	0.00858
Population	Latitude	0.0847	-0.0627
Population	Longitude	0.0835	0.0591
HouseAge	noise_cat	0.0820	0.00306
Population	noise_cat	0.0808	-0.0349
Longitude	noise_cont	0.0805	-0.0212
AveBedrms	noise_cont	0.0795	0.0398
MedInc	Longitude	0.0794	-0.0368
HouseAge	AveBedrms	0.0754	-0.0916
MedInc	noise_cat	0.0741	0.0127
MedInc	Population	0.0712	-0.0118
MedInc	AveOccup	0.0711	-0.0681
AveRooms	noise_cont	0.0708	0.0495
Latitude	noise_cont	0.0677	0.0270
Population	noise_cont	0.0676	-0.00111
HouseAge	noise_cont	0.0669	0.0305
MedInc	noise_cont	0.0669	0.00776
MedInc	AveBedrms	0.0512	-0.0627
AveRooms	Population	0.0420	-0.104
AveRooms	AveOccup	0.0324	-0.0654
AveBedrms	AveOccup	0.0284	-0.0771
AveBedrms	Population	0.0272	-0.0773

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

.. GENERATED FROM PYTHON SOURCE LINES 74-76 .. code-block:: Python TableReport(y) .. raw:: html

	MedHouseVal
1,763	1.34
18,259	3.73
13,547	0.709
11,536	0.675
15,069	1.36

11,284	2.29
11,964	0.978
5,390	2.22
860	2.83
15,795	3.25

Column	Column name	dtype	Is sorted	Null values	Unique values	Mean	Std	Min	Median	Max
0	MedHouseVal	Float64DType	False	0 (0.0%)	1072 (71.5%)	2.07	1.16	0.250	1.79	5.00

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

.. GENERATED FROM PYTHON SOURCE LINES 77-78 Counting unique values per column previews which features SKD007 will flag. .. GENERATED FROM PYTHON SOURCE LINES 78-81 .. code-block:: Python X.nunique().sort_values(ascending=False) .. rst-class:: sphx-glr-script-out .. code-block:: none noise_cont 1500 AveRooms 1491 AveOccup 1478 AveBedrms 1408 MedInc 1380 Population 1128 Longitude 483 Latitude 458 HouseAge 51 noise_cat 20 dtype: int64 .. GENERATED FROM PYTHON SOURCE LINES 82-87 Trigger SKD007 with a random forest on continuous features ========================================================== A random forest exposes ``feature_importances_`` based on MDI. After fitting, let us inspect impurity decrease with skore. .. GENERATED FROM PYTHON SOURCE LINES 87-101 .. code-block:: Python from sklearn.ensemble import RandomForestRegressor from skore import TrainTestSplit, evaluate splitter = TrainTestSplit(random_state=42) report = evaluate( RandomForestRegressor(random_state=42), X=X, y=y, splitter=splitter, ) report .. raw:: html

Metric	RandomForestRegressor
R²	0.737305
RMSE	0.610720
MAE	0.413099
MAPE	0.242566
Fit time (s)	0.574603
Predict time (s)	0.017045

RandomForestRegressor(random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

RandomForestRegressor

?Documentation for RandomForestRegressoriFitted

Parameters

	random_state random_state: int, RandomState instance or None, default=None Controls both the randomness of the bootstrapping of the samples used when building trees (if ``bootstrap=True``) and the sampling of the features to consider when looking for the best split at each node (if ``max_features < n_features``). See :term:`Glossary <random_state>` for details.	42
	n_estimators n_estimators: int, default=100 The number of trees in the forest. .. versionchanged:: 0.22 The default value of ``n_estimators`` changed from 10 to 100 in 0.22.	100
	criterion criterion: {"squared_error", "absolute_error", "poisson"}, default="squared_error" The function to measure the quality of a split. Supported criteria are "squared_error" for the mean squared error, which is equal to variance reduction as feature selection criterion and minimizes the L2 loss using the mean of each terminal node, "absolute_error" for the mean absolute error, which minimizes the L1 loss using the median of each terminal node, and "poisson" which uses reduction in Poisson deviance to find splits, also using the mean of each terminal node. .. versionadded:: 0.18 Mean Absolute Error (MAE) criterion. .. versionadded:: 1.0 Poisson criterion. .. versionchanged:: 1.9 Criterion `"friedman_mse"` was deprecated.	'squared_error'
	max_depth max_depth: int, default=None The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.	None
	min_samples_split min_samples_split: int or float, default=2 The minimum number of samples required to split an internal node: - If int, then consider `min_samples_split` as the minimum number. - If float, then `min_samples_split` is a fraction and `ceil(min_samples_split * n_samples)` are the minimum number of samples for each split. .. versionchanged:: 0.18 Added float values for fractions.	2
	min_samples_leaf min_samples_leaf: int or float, default=1 The minimum number of samples required to be at a leaf node. A split point at any depth will only be considered if it leaves at least ``min_samples_leaf`` training samples in each of the left and right branches. This may have the effect of smoothing the model, especially in regression. - If int, then consider `min_samples_leaf` as the minimum number. - If float, then `min_samples_leaf` is a fraction and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node. .. versionchanged:: 0.18 Added float values for fractions.	1
	min_weight_fraction_leaf min_weight_fraction_leaf: float, default=0.0 The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided.	0.0
	max_features max_features: {"sqrt", "log2", None}, int or float, default=1.0 The number of features to consider when looking for the best split: - If int, then consider `max_features` features at each split. - If float, then `max_features` is a fraction and `max(1, int(max_features * n_features_in_))` features are considered at each split. - If "sqrt", then `max_features=sqrt(n_features)`. - If "log2", then `max_features=log2(n_features)`. - If None or 1.0, then `max_features=n_features`. .. note:: The default of 1.0 is equivalent to bagged trees and more randomness can be achieved by setting smaller values, e.g. 0.3. .. versionchanged:: 1.1 The default of `max_features` changed from `"auto"` to 1.0. Note: the search for a split does not stop until at least one valid partition of the node samples is found, even if it requires to effectively inspect more than ``max_features`` features.	1.0
	max_leaf_nodes max_leaf_nodes: int, default=None Grow trees with ``max_leaf_nodes`` in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.	None
	min_impurity_decrease min_impurity_decrease: float, default=0.0 A node will be split if this split induces a decrease of the impurity greater than or equal to this value. The weighted impurity decrease equation is the following:: N_t / N * (impurity - N_t_R / N_t * right_impurity - N_t_L / N_t * left_impurity) where ``N`` is the total number of samples, ``N_t`` is the number of samples at the current node, ``N_t_L`` is the number of samples in the left child, and ``N_t_R`` is the number of samples in the right child. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum, if ``sample_weight`` is passed. .. versionadded:: 0.19	0.0
	bootstrap bootstrap: bool, default=True Whether bootstrap samples are used when building trees. If False, the whole dataset is used to build each tree.	True
	oob_score oob_score: bool or callable, default=False Whether to use out-of-bag samples to estimate the generalization score. By default, :func:`~sklearn.metrics.r2_score` is used. Provide a callable with signature `metric(y_true, y_pred)` to use a custom metric. Only available if `bootstrap=True`. For an illustration of out-of-bag (OOB) error estimation, see the example :ref:`sphx_glr_auto_examples_ensemble_plot_ensemble_oob.py`.	False
	n_jobs n_jobs: int, default=None The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`, :meth:`decision_path` and :meth:`apply` are all parallelized over the trees. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary <n_jobs>` for more details.	None
	verbose verbose: int, default=0 Controls the verbosity when fitting and predicting.	0
	warm_start warm_start: bool, default=False When set to ``True``, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new forest. See :term:`Glossary <warm_start>` and :ref:`tree_ensemble_warm_start` for details.	False
	ccp_alpha ccp_alpha: non-negative float, default=0.0 Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ``ccp_alpha`` will be chosen. By default, no pruning is performed. See :ref:`minimal_cost_complexity_pruning` for details. See :ref:`sphx_glr_auto_examples_tree_plot_cost_complexity_pruning.py` for an example of such pruning. .. versionadded:: 0.22	0.0
	max_samples max_samples: int or float, default=None If bootstrap is True, the number of samples to draw from X to train each base estimator. - If None (default), then draw `X.shape[0]` samples irrespective of `sample_weight`. - If int, then draw `max_samples` samples. - If float, then draw `max_samples * X.shape[0]` unweighted samples or `max_samples * sample_weight.sum()` weighted samples. .. versionadded:: 0.22 .. versionchanged:: 1.9 Float `max_samples` is relative to `sample_weight.sum()` instead of `X.shape[0]` for weighted samples.	None
	monotonic_cst monotonic_cst: array-like of int of shape (n_features), default=None Indicates the monotonicity constraint to enforce on each feature. - 1: monotonically increasing - 0: no constraint - -1: monotonically decreasing If monotonic_cst is None, no constraints are applied. Monotonicity constraints are not supported for: - multioutput regressions (i.e. when `n_outputs_ > 1`). Read more in the :ref:`User Guide <monotonic_cst_gbdt>`. .. versionadded:: 1.4	None

Fitted attributes

Name	Type	Value
estimator_ estimator_: :class:`~sklearn.tree.DecisionTreeRegressor` The child estimator template used to create the collection of fitted sub-estimators. .. versionadded:: 1.2 `base_estimator_` was renamed to `estimator_`.	DecisionTreeRegressor	DecisionTreeRegressor()
estimators_ estimators_: list of DecisionTreeRegressor The collection of fitted sub-estimators.	list	[DecisionTreeR...te=1608637542), DecisionTreeR...te=1273642419), DecisionTreeR...te=1935803228), DecisionTreeR...ate=787846414), ...]
estimators_samples_ estimators_samples_: list of arrays The subset of drawn samples (i.e., the in-bag samples) for each base estimator. Each subset is defined by an array of the indices selected. .. versionadded:: 1.4	list	[array([ 475, ..., dtype=int32), array([ 686, ..., dtype=int32), array([ 430, ..., dtype=int32), array([800, 5..., dtype=int32), ...]
feature_importances_ feature_importances_: ndarray of shape (n_features,) The impurity-based feature importances. The higher, the more important the feature. The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance. Warning: impurity-based feature importances can be misleading for high cardinality features (many unique values). See :func:`sklearn.inspection.permutation_importance` as an alternative.	ndarray[float64](10,)	[0.52,0.07,0.05,...,0.05,0.03,0.02]
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0	ndarray[object](10,)	['MedInc','HouseAge','AveRooms',...,'Longitude','noise_cont','noise_cat']
n_features_in_ n_features_in_: int Number of features seen during :term:`fit`. .. versionadded:: 0.24	int	10
n_outputs_ n_outputs_: int The number of outputs when ``fit`` is performed.	int	1

	MedInc	HouseAge	AveRooms	AveBedrms	Population	AveOccup	Latitude	Longitude	noise_cont	noise_cat	MedHouseVal
0	3.69	33.0	6.56	1.23	652.	2.85	38.6	-121.	-1.60	17	0.993
1	3.44	22.0	5.79	1.02	1.20e+03	3.24	39.1	-122.	-2.39	8	1.02
2	4.46	40.0	5.48	0.936	756.	2.68	37.6	-122.	-1.07	17	3.08
3	1.55	52.0	5.02	1.00	311.	3.11	39.7	-122.	-1.15	14	2.00
4	4.41	23.0	5.67	0.989	1.58e+03	2.96	34.0	-117.	1.82	7	1.30

1,495	2.31	52.0	2.70	1.09	679.	1.48	34.0	-119.	0.736	10	5.00
1,496	2.91	27.0	3.80	1.12	2.01e+03	3.67	33.9	-118.	0.0389	0	1.84
1,497	2.52	28.0	3.79	1.07	1.36e+03	4.76	34.0	-118.	-0.115	10	1.32
1,498	4.68	16.0	5.35	1.09	2.53e+03	3.10	37.6	-122.	0.184	1	2.56
1,499	2.30	17.0	4.93	1.13	2.02e+03	3.00	33.8	-117.	-1.86	17	0.893

Column	Column name	dtype	Is sorted	Unique values	Mean	Std	Min	Median	Max
0	MedInc	Float64DType	False	1380 (92.0%)	3.86	1.89	0.536	3.51	15.0
1	HouseAge	Float64DType	False	51 (3.4%)	28.3	12.5	2.00	28.0	52.0
2	AveRooms	Float64DType	False	1491 (99.4%)	5.43	2.17	1.63	5.26	47.5
3	AveBedrms	Float64DType	False	1408 (93.9%)	1.10	0.421	0.810	1.05	11.2
4	Population	Float64DType	False	1128 (75.2%)	1.43e+03	1.07e+03	3.00	1.17e+03	1.22e+04
5	AveOccup	Float64DType	False	1478 (98.5%)	2.94	0.921	0.750	2.83	21.3
6	Latitude	Float64DType	False	458 (30.5%)	35.5	2.09	32.6	34.2	41.8
7	Longitude	Float64DType	False	483 (32.2%)	-119.	1.97	-124.	-118.	-114.
8	noise_cont	Float64DType	False	1500 (100.0%)	-0.0206	1.00	-3.65	0.00380	3.18
9	noise_cat	Int64DType	False	20 (1.3%)	9.39	5.76	0	9	19
10	MedHouseVal	Float64DType	False	1072 (71.5%)	2.07	1.16	0.250	1.79	5.00

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

.. GENERATED FROM PYTHON SOURCE LINES 102-105 SKD007 warns about high-cardinality columns such as ``MedInc`` and ``AveOccup``. The synthetic continuous noise column is high-cardinality as well, so it belongs in the same tip. .. GENERATED FROM PYTHON SOURCE LINES 105-108 .. code-block:: Python report.checks.summarize(fast_mode=True) .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 109-110 Let us plot MDI with features sorted by importance. .. GENERATED FROM PYTHON SOURCE LINES 110-116 .. code-block:: Python import matplotlib.pyplot as plt mdi_display = report.inspection.impurity_decrease() _ = mdi_display.plot(sorting_order="descending") .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd007_mdi_cardinality_bias_001.png :alt: Mean decrease in impurity (MDI) of RandomForestRegressor :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd007_mdi_cardinality_bias_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 117-120 The two synthetic noise columns are not near zero under MDI: impurity still assigns them mass. ``noise_cont`` in particular is high-cardinality, so the forest can keep finding splits on it even though it carries no target signal. .. GENERATED FROM PYTHON SOURCE LINES 122-128 Use permutation importance instead of MDI ========================================= :meth:`~skore.EstimatorReport.inspection.permutation_importance` shuffles each column on the test set and measures the score drop. The result is not biased toward high-cardinality split points the way MDI is. .. GENERATED FROM PYTHON SOURCE LINES 128-135 .. code-block:: Python perm_display = report.inspection.permutation_importance( seed=42, n_repeats=5, ) _ = perm_display.plot(sorting_order="descending") .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd007_mdi_cardinality_bias_002.png :alt: Permutation importance of RandomForestRegressor on test set :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd007_mdi_cardinality_bias_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 136-139 Under permutation importance the noisy features sit at (or very near) zero: shuffling them does not change the test score, so they are not contributing to predicting the target. .. GENERATED FROM PYTHON SOURCE LINES 141-148 Cross-check MDI with permutation importance =========================================== Let us put the two rankings side by side. Sorting by MDI tends to push high-cardinality columns (including ``noise_cont``) upward; permutation importance on the test set should keep both synthetic features near the bottom even when MDI does not. .. GENERATED FROM PYTHON SOURCE LINES 148-172 .. code-block:: Python mdi = ( mdi_display.frame(sorting_order="descending") .set_index("feature") .rename(columns={"importance": "mdi"}) ) perm = ( perm_display.frame(sorting_order="descending") .set_index("feature")[["value_mean"]] .rename(columns={"value_mean": "permutation"}) ) nunique = X.nunique().rename("nunique") comparison = ( mdi.join(perm) .join(nunique) .assign( mdi_rank=lambda df: df["mdi"].rank(ascending=False).astype(int), perm_rank=lambda df: df["permutation"].rank(ascending=False).astype(int), ) .sort_values("mdi", ascending=False) ) comparison .. raw:: html

	mdi	permutation	nunique	mdi_rank	perm_rank
feature
MedInc	0.522390	1.134807	1380	1	1
AveOccup	0.140019	0.128296	1478	2	2
HouseAge	0.067881	0.050188	51	3	5
Longitude	0.054675	0.082798	483	4	4
Latitude	0.050791	0.128265	458	5	3
AveRooms	0.049150	0.044559	1491	6	6
AveBedrms	0.037725	0.014047	1408	7	7
Population	0.031908	0.011334	1128	8	8
noise_cont	0.028676	0.004603	1500	9	9
noise_cat	0.016786	0.001149	20	10	10

.. GENERATED FROM PYTHON SOURCE LINES 173-176 The side-by-side bars make the disagreement easier to read: impurity can assign mass to ``noise_cont`` (and sometimes more than to ``noise_cat``), while permutation importance stays close to zero for both. .. GENERATED FROM PYTHON SOURCE LINES 176-191 .. code-block:: Python fig, axes = plt.subplots(1, 2, figsize=(12, 5), sharey=True) order = comparison.sort_values("mdi", ascending=True).index axes[0].barh(order, comparison.loc[order, "mdi"]) axes[0].set_title("MDI (impurity decrease)") axes[0].set_xlabel("Importance") axes[1].barh(order, comparison.loc[order, "permutation"]) axes[1].set_title("Permutation importance (test)") axes[1].set_xlabel("Mean score drop") fig.tight_layout() _ = fig .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd007_mdi_cardinality_bias_003.png :alt: MDI (impurity decrease), Permutation importance (test) :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd007_mdi_cardinality_bias_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 192-200 Conclusion ========== SKD007 warns that MDI feature importance favors high-cardinality inputs such as ``AveOccup`` and can inflate the role of irrelevant high-cardinality noise. In this walkthrough, permutation importance gave a more reliable picture of which features actually move test scores. When importance is a decision factor, we prefer permutation (or drop-column tests) over impurity alone. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 4.958 seconds) .. _sphx_glr_download_auto_examples_pitfalls_and_solutions_plot_skd007_mdi_cardinality_bias.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_skd007_mdi_cardinality_bias.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_skd007_mdi_cardinality_bias.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_skd007_mdi_cardinality_bias.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_

MedInc

HouseAge

AveRooms

AveBedrms

Population

AveOccup

Latitude

Longitude

noise_cont

noise_cat

MedInc

HouseAge

AveRooms

AveBedrms

Population

AveOccup

Latitude

Longitude

noise_cont

noise_cat

Please enable javascript

MedHouseVal

MedHouseVal

Please enable javascript

MedInc

HouseAge

AveRooms

AveBedrms

Population

AveOccup

Latitude

Longitude

noise_cont

noise_cat

MedHouseVal

Please enable javascript