EstimatorReport.metrics.custom_metric#

EstimatorReport.metrics.custom_metric(metric_function, response_method, *, data_source='test', X=None, y=None, **kwargs)[source]#

Compute a custom metric.

It brings some flexibility to compute any desired metric. However, we need to follow some rules:

  • metric_function should take y_true and y_pred as the first two positional arguments.

  • response_method corresponds to the estimator’s method to be invoked to get the predictions. It can be a string or a list of strings to defined in which order the methods should be invoked.

Parameters:
metric_functioncallable

The metric function to be computed. The expected signature is metric_function(y_true, y_pred, **kwargs).

response_methodstr or list of str

The estimator’s method to be invoked to get the predictions. The possible values are: predict, predict_proba, predict_log_proba, and decision_function.

data_source{“test”, “train”, “X_y”}, default=”test”

The data source to use.

  • “test” : use the test set provided when creating the report.

  • “train” : use the train set provided when creating the report.

  • “X_y” : use the provided X and y to compute the metric.

Xarray-like of shape (n_samples, n_features), default=None

New data on which to compute the metric. By default, we use the validation set provided when creating the report.

yarray-like of shape (n_samples,), default=None

New target on which to compute the metric. By default, we use the target provided when creating the report.

**kwargsdict

Any additional keyword arguments to be passed to the metric function.

Returns:
float, dict, or list of n_outputs

The custom metric. The output type depends on the metric function.

Examples

>>> from sklearn.datasets import load_diabetes
>>> from sklearn.linear_model import Ridge
>>> from sklearn.metrics import mean_absolute_error
>>> from sklearn.model_selection import train_test_split
>>> from skore import EstimatorReport
>>> X_train, X_test, y_train, y_test = train_test_split(
...     *load_diabetes(return_X_y=True), random_state=0
... )
>>> regressor = Ridge()
>>> report = EstimatorReport(
...     regressor,
...     X_train=X_train,
...     y_train=y_train,
...     X_test=X_test,
...     y_test=y_test,
... )
>>> report.metrics.custom_metric(
...     metric_function=mean_absolute_error,
...     response_method="predict",
... )
44.9...
>>> def metric_function(y_true, y_pred):
...     return {"output": float(mean_absolute_error(y_true, y_pred))}
>>> report.metrics.custom_metric(
...     metric_function=metric_function,
...     response_method="predict",
... )
{'output': 44.9...}