EstimatorReport.inspection.permutation_importance#
- EstimatorReport.inspection.permutation_importance(*, data_source='test', X=None, y=None, at_step=0, metric=None, n_repeats=5, max_samples=1.0, n_jobs=None, seed=None)[source]#
Report the permutation feature importance.
This computes the permutation importance using sklearn’s
permutation_importance()function, which consists in permuting the values of one feature and comparing the value ofmetricbetween with and without the permutation, which gives an indication on the impact of the feature.By default,
seedis set toNone, which means the function will return a different result at every call. In that case, the results are not cached. If you wish to take advantage of skore’s caching capabilities, make sure you set theseedparameter.- Parameters:
- data_source{“test”, “train”, “X_y”}, default=”test”
The data source to use.
“test” : use the test set provided when creating the report.
“train” : use the train set provided when creating the report.
“X_y” : use the provided
Xandyto compute the metric.
- Xarray-like of shape (n_samples, n_features), default=None
New data on which to compute the metric. By default, we use the test set provided when creating the report.
- yarray-like of shape (n_samples,), default=None
New target on which to compute the metric. By default, we use the test target provided when creating the report.
- at_stepint or str, default=0
If the estimator is a
Pipeline, at which step of the pipeline the importance is computed. Ifn, then the features that are evaluated are the ones right before then-th step of the pipeline. For instance,If 0, compute the importance just before the start of the pipeline (i.e. the importance of the raw input features).
If -1, compute the importance just before the end of the pipeline (i.e. the importance of the fully engineered features, just before the actual prediction step).
If a string, will be searched among the pipeline’s
named_steps.Has no effect if the estimator is not a
Pipeline.- metricstr, callable, list, tuple, or dict, default=None
The metric to pass to
permutation_importance().If
metricrepresents a single metric, one can use:a single string, which must be one of the supported metrics;
a callable that returns a single value.
If
metricrepresents multiple metrics, one can use:a list or tuple of unique strings, which must be one of the supported metrics;
a callable returning a dictionary where the keys are the metric names and the values are the metric scores;
a dictionary with metric names as keys and callables a values.
- n_repeatsint, default=5
Number of times to permute a feature.
- max_samplesint or float, default=1.0
The number of samples to draw from
Xto compute feature importance in each repeat (without replacement).If int, then draw max_samples samples.
If float, then draw max_samples * X.shape[0] samples.
If max_samples is equal to 1.0 or X.shape[0], all samples will be used.
While using this option may provide less accurate importance estimates, it keeps the method tractable when evaluating feature importance on large datasets. In combination with n_repeats, this allows to control the computational speed vs statistical accuracy trade-off of this method.
- n_jobsint or None, default=None
Number of jobs to run in parallel. -1 means using all processors.
- seedint or None, default=None
The seed used to initialize the random number generator used for the permutations.
- Returns:
- pandas.DataFrame
The permutation importance.
Notes
Even if pipeline components output sparse arrays, these will be made dense.
Examples
>>> from sklearn.datasets import make_regression >>> from sklearn.linear_model import Ridge >>> from skore import train_test_split >>> from skore import EstimatorReport >>> X, y = make_regression(n_features=3, random_state=0) >>> split_data = train_test_split(X=X, y=y, random_state=0, as_dict=True) >>> regressor = Ridge() >>> report = EstimatorReport(regressor, **split_data)
>>> report.inspection.permutation_importance( ... n_repeats=2, ... seed=0, ... ).frame(aggregate=None) data_source metric feature repetition value 0 test r2 Feature #0 1 0.69... 1 test r2 Feature #1 1 2.32... 2 test r2 Feature #2 1 0.02... 3 test r2 Feature #0 2 0.88... 4 test r2 Feature #1 2 2.63... 5 test r2 Feature #2 2 0.02...
>>> report.inspection.permutation_importance( ... metric=["r2", "neg_mean_squared_error"], ... n_repeats=2, ... seed=0, ... ).frame(aggregate=None) data_source metric feature repetition value 0 test r2 Feature #0 1 0.69... 1 test r2 Feature #1 1 2.32... 2 test r2 Feature #2 1 0.02... 3 test r2 Feature #0 2 0.88... 4 test r2 Feature #1 2 2.63... 5 test r2 Feature #2 2 0.02... 0 test neg_mean_squared_error Feature #0 1 2298.79... 1 test neg_mean_squared_error Feature #1 1 7627.28... 2 test neg_mean_squared_error Feature #2 1 92.78... 3 test neg_mean_squared_error Feature #0 2 2911.23... 4 test neg_mean_squared_error Feature #1 2 8666.16... 5 test neg_mean_squared_error Feature #2 2 74.21...
>>> report.inspection.permutation_importance( ... n_repeats=2, ... seed=0, ... ).frame() data_source metric feature value_mean value_std 0 test r2 Feature #0 0.79... 0.13... 1 test r2 Feature #1 2.47... 0.22... 2 test r2 Feature #2 0.02... 0.00...
>>> # Compute the importance at the end of feature engineering pipeline >>> from sklearn.pipeline import make_pipeline >>> from sklearn.preprocessing import StandardScaler >>> pipeline = make_pipeline(StandardScaler(), Ridge()) >>> pipeline_report = EstimatorReport(pipeline, **split_data) >>> pipeline_report.inspection.permutation_importance( ... n_repeats=2, ... seed=0, ... at_step=-1, ... ).frame() data_source metric feature value_mean value_std 0 test r2 x0 0.79... 0.13... 1 test r2 x1 2.47... 0.22... 2 test r2 x2 0.02... 0.00...
>>> pipeline_report.inspection.permutation_importance( ... n_repeats=2, ... seed=0, ... at_step="ridge", ... ).frame() data_source metric feature value_mean value_std 0 test r2 x0 0.79... 0.13... 1 test r2 x1 2.47... 0.22... 2 test r2 x2 0.02... 0.00...