permutation_importance#
- ComparisonReport.inspection.permutation_importance(*, data_source='test', at_step=0, metric=None, n_repeats=5, max_samples=1.0, n_jobs=None, seed=None)[source]#
Display the permutation feature importance.
This computes the permutation importance using sklearn’s
permutation_importance()function, which consists in permuting the values of one feature and comparing the value ofmetricbetween with and without the permutation, which gives an indication on the impact of the feature.By default,
seedis set toNone, which means the function will return a different result at every call. In that case, the results are not cached. If you wish to take advantage of skore’s caching capabilities, make sure you set theseedparameter.Comparison reports with the same features are put under one key and are plotted together. When some reports share the same features and others do not, those with the same features are plotted together.
- Parameters:
- data_source{“test”, “train”}, default=”test”
The data source to use.
“test” : use the test set provided when creating the report.
“train” : use the train set provided when creating the report.
- at_stepint or str, default=0
If the estimator is a
Pipeline, at which step of the pipeline the importance is computed. Ifn, then the features that are evaluated are the ones right before then-th step of the pipeline. For instance,If 0, compute the importance just before the start of the pipeline (i.e. the importance of the raw input features).
If -1, compute the importance just before the end of the pipeline (i.e. the importance of the fully engineered features, just before the actual prediction step).
If a string, will be searched among the pipeline’s
named_steps.Has no effect if the estimator is not a
Pipeline.- metricstr, callable, scorer, or list of such instances or dict of such instances, default=None
The metric to pass to
permutation_importance(). The possible values (whether or not in a list) are:if a string, either one of the built-in metrics or a scikit-learn scorer name. You can get the possible list of string using
report.metrics.help()orsklearn.metrics.get_scorer_names()for the built-in metrics or the scikit-learn scorers, respectively.if a callable, it should take as arguments
y_true,y_predas the two first arguments. Additional arguments can be passed as keyword arguments and will be forwarded withmetric_kwargs. No favorability indicator can be displayed in this case.if the callable API is too restrictive (e.g. need to pass same parameter name with different values), you can use scikit-learn scorers as provided by
sklearn.metrics.make_scorer(). In this case, the metric favorability will only be displayed if it is given explicitly viamake_scorer’sgreater_is_betterparameter.
- n_repeatsint, default=5
Number of times to permute a feature.
- max_samplesint or float, default=1.0
The number of samples to draw from
Xto compute feature importance in each repeat (without replacement).If int, then draw max_samples samples.
If float, then draw max_samples * X.shape[0] samples.
If max_samples is equal to 1.0 or X.shape[0], all samples will be used.
While using this option may provide less accurate importance estimates, it keeps the method tractable when evaluating feature importance on large datasets. In combination with n_repeats, this allows to control the computational speed vs statistical accuracy trade-off of this method.
- n_jobsint or None, default=None
Number of jobs to run in parallel. -1 means using all processors.
- seedint or None, default=None
The seed used to initialize the random number generator used for the permutations.
- Returns:
PermutationImportanceDisplayThe permutation importance display.
Notes
Even if pipeline components output sparse arrays, these will be made dense.
Examples
>>> from sklearn.datasets import make_regression >>> from sklearn.linear_model import Ridge >>> from sklearn.metrics import make_scorer, r2_score, mean_squared_error >>> from skore import train_test_split >>> from skore import ComparisonReport, EstimatorReport >>> X, y = make_regression(n_features=3, random_state=0) >>> split_data = train_test_split(X=X, y=y, shuffle=False, as_dict=True) >>> report_big_alpha = EstimatorReport(Ridge(alpha=1e3), **split_data) >>> report_small_alpha = EstimatorReport(Ridge(alpha=1e-3), **split_data) >>> report = ComparisonReport({ ... "small alpha": report_small_alpha, ... "big alpha": report_big_alpha, ... }) >>> report.inspection.permutation_importance( ... n_repeats=2, ... seed=0, ... ).frame(aggregate=None) estimator data_source metric feature repetition value 0 small alpha test r2 Feature #0 1 0.41... 1 small alpha test r2 Feature #1 1 1.16... 2 small alpha test r2 Feature #2 1 0.01... 3 small alpha test r2 Feature #0 2 0.73... 4 small alpha test r2 Feature #1 2 0.90... 5 small alpha test r2 Feature #2 2 0.01... 6 big alpha test r2 Feature #0 1 0.03... 7 big alpha test r2 Feature #1 1 0.10... 8 big alpha test r2 Feature #2 1 0.00... 9 big alpha test r2 Feature #0 2 0.05... 10 big alpha test r2 Feature #1 2 0.07... 11 big alpha test r2 Feature #2 2 -0.00...
>>> report.inspection.permutation_importance( ... metric={ ... "r2": make_scorer(r2_score), ... "mse": make_scorer(mean_squared_error), ... }, ... n_repeats=2, ... seed=0, ... ).frame() estimator data_source metric ... value_mean value_std 0 small alpha test r2 ... 0.577888 0.226399 1 small alpha test r2 ... 1.035454 0.180778 2 small alpha test r2 ... 0.016801 0.003577 3 small alpha test mse ... 3749.565731 1468.964660 4 small alpha test mse ... 6718.437884 1172.961272 5 small alpha test mse ... 109.009650 23.205795 6 big alpha test r2 ... 0.044482 0.013717 7 big alpha test r2 ... 0.088916 0.026191 8 big alpha test r2 ... 0.001826 0.008475 9 big alpha test mse ... 288.618702 88.998809 10 big alpha test mse ... 576.924696 169.940308 11 big alpha test mse ... 11.846779 54.992092