Adapt skore to your use-case by adding your own metrics

By default, summarize() reports a curated set of metrics for your ML task. In practice you often need domain-specific scores: a business cost function, a custom fairness measure, an F-beta with a particular beta, etc.

This example walks through how to register such metrics with add() so they are computed and displayed alongside the built-in ones.

Setting up a classification problem

import skore
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)

We create an EstimatorReport through evaluate() using a simple train/test split. pos_label=1 marks the malignant class as the positive class.

report = skore.evaluate(
    LogisticRegression(max_iter=10_000), X, y, pos_label=1, splitter=0.2
)

Let’s look at the default metrics:

report.metrics.summarize().frame()

metric
accuracy        0.947368
precision       0.984127
recall          0.925373
roc_auc         0.993649
log_loss        0.110247
brier_score     0.036154
fit_time        0.238488
predict_time    0.000131
Name: LogisticRegression, dtype: float64

Adding a plain callable

Any function with the signature (estimator, X, y, **kwargs) -> score can be registered with add(). The function name is used as the metric name by default. If your metric can be expressed as a callable with the signature (y_true, y_pred, **kwargs) -> score, then you can use sklearn’s make_scorer utility function to convert it.

from sklearn.metrics import make_scorer


def specificity(y_true, y_pred):
    """Proportion of true negatives among actual negatives."""
    tn = ((y_true == 0) & (y_pred == 0)).sum()
    fp = ((y_true == 0) & (y_pred == 1)).sum()
    return tn / (tn + fp)


report.metrics.add(make_scorer(specificity))

report.metrics.summarize().frame()

metric
specificity     0.978723
accuracy        0.947368
precision       0.984127
recall          0.925373
roc_auc         0.993649
log_loss        0.110247
brier_score     0.036154
fit_time        0.238488
predict_time    0.000131
Name: LogisticRegression, dtype: float64

specificity now appears alongside the built-in metrics. It can also be computed individually using the get() method:

report.metrics.get("specificity")

0.9787234042553191

Further, if the metric name is a valid identifier, the metric is also exposed as a method:

report.metrics.specificity()

0.9787234042553191

Passing extra keyword arguments

If your metric needs extra data at scoring time (e.g. sample-level amounts, a cost matrix, …), they can be passed as keyword arguments to add(); they will be forwarded to the metric function when it is computed. Alternatively, if the metric takes y_true and y_pred, the keyword arguments can be passed to make_scorer:

from sklearn.metrics import fbeta_score, make_scorer

f2_scorer = make_scorer(fbeta_score, beta=2, pos_label=1)
report.metrics.add(f2_scorer, name="f2")

report.metrics.summarize().frame()

metric
f2              0.936556
specificity     0.978723
accuracy        0.947368
precision       0.984127
recall          0.925373
roc_auc         0.993649
log_loss        0.110247
brier_score     0.036154
fit_time        0.238488
predict_time    0.000131
Name: LogisticRegression, dtype: float64

Cherry-picking metrics to display

Once registered, custom metrics can be selected by name in summarize():

report.metrics.summarize(metric=["specificity", "f2"]).frame()

metric
specificity    0.978723
f2             0.936556
Name: LogisticRegression, dtype: float64

Selecting data_source="both" lets you compare train vs. test in one call:

report.metrics.summarize(metric=["specificity", "f2"], data_source="both").frame()

	LogisticRegression (train)	LogisticRegression (test)
metric
specificity	0.933333	0.978723
f2	0.975945	0.936556

Using a different response method

By default, callables receive the output of estimator.predict(X). If your metric needs probabilities instead, set response_method="predict_proba".

import numpy as np


def mean_confidence(y_true, y_proba):
    """Average predicted probability assigned to the true class."""
    return np.where(y_true == 1, y_proba[:, 1], y_proba[:, 0]).mean()


report.metrics.add(make_scorer(mean_confidence, response_method="predict_proba"))

report.metrics.summarize(metric="mean_confidence").frame()

metric
mean_confidence    0.931087
Name: LogisticRegression, dtype: float64