.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/model_evaluation/plot_custom_metrics.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_model_evaluation_plot_custom_metrics.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_model_evaluation_plot_custom_metrics.py:


.. _example_custom_metrics:

===========================================================
`EstimatorReport.metrics.add`: Adapt skore to your use-case
===========================================================

By default, :meth:`~skore.EstimatorReport.metrics.summarize` reports a curated
set of metrics for your ML task. In practice you often need domain-specific
scores: a business cost function, a custom fairness measure, an F-beta with a
particular beta, etc.

This example walks through how to register such metrics with
:meth:`~skore.EstimatorReport.metrics.add` so they are computed and displayed
alongside the built-in ones.

.. GENERATED FROM PYTHON SOURCE LINES 19-21

Setting up a classification problem
===================================

.. GENERATED FROM PYTHON SOURCE LINES 23-29

.. code-block:: Python

    import skore
    from sklearn.datasets import load_breast_cancer
    from sklearn.linear_model import LogisticRegression

    X, y = load_breast_cancer(return_X_y=True)








.. GENERATED FROM PYTHON SOURCE LINES 30-33

We create an :class:`~skore.EstimatorReport` through :func:`~skore.evaluate`
using a simple train/test split. ``pos_label=1`` marks the *malignant* class
as the positive class.

.. GENERATED FROM PYTHON SOURCE LINES 35-39

.. code-block:: Python

    report = skore.evaluate(
        LogisticRegression(max_iter=10_000), X, y, pos_label=1, splitter=0.2
    )








.. GENERATED FROM PYTHON SOURCE LINES 40-41

Let's look at the default metrics:

.. GENERATED FROM PYTHON SOURCE LINES 41-43

.. code-block:: Python

    report.metrics.summarize().frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>LogisticRegression</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Accuracy</th>
          <td>0.947368</td>
        </tr>
        <tr>
          <th>Precision</th>
          <td>0.984127</td>
        </tr>
        <tr>
          <th>Recall</th>
          <td>0.925373</td>
        </tr>
        <tr>
          <th>ROC AUC</th>
          <td>0.993649</td>
        </tr>
        <tr>
          <th>Log loss</th>
          <td>0.110103</td>
        </tr>
        <tr>
          <th>Brier score</th>
          <td>0.036139</td>
        </tr>
        <tr>
          <th>Fit time (s)</th>
          <td>0.199774</td>
        </tr>
        <tr>
          <th>Predict time (s)</th>
          <td>0.000118</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 44-50

Adding a plain callable
=======================

Any function with the signature ``(y_true, y_pred, **kwargs) -> float`` can be
registered with :meth:`~skore.EstimatorReport.metrics.add`. The function name
is used as the metric name by default.

.. GENERATED FROM PYTHON SOURCE LINES 53-62

.. code-block:: Python

    def specificity(y_true, y_pred):
        """Proportion of true negatives among actual negatives."""
        tn = ((y_true == 0) & (y_pred == 0)).sum()
        fp = ((y_true == 0) & (y_pred == 1)).sum()
        return tn / (tn + fp)


    report.metrics.add(specificity, greater_is_better=True)








.. GENERATED FROM PYTHON SOURCE LINES 63-65

.. code-block:: Python

    report.metrics.summarize().frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>LogisticRegression</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Accuracy</th>
          <td>0.947368</td>
        </tr>
        <tr>
          <th>Precision</th>
          <td>0.984127</td>
        </tr>
        <tr>
          <th>Recall</th>
          <td>0.925373</td>
        </tr>
        <tr>
          <th>ROC AUC</th>
          <td>0.993649</td>
        </tr>
        <tr>
          <th>Log loss</th>
          <td>0.110103</td>
        </tr>
        <tr>
          <th>Brier score</th>
          <td>0.036139</td>
        </tr>
        <tr>
          <th>Fit time (s)</th>
          <td>0.199774</td>
        </tr>
        <tr>
          <th>Predict time (s)</th>
          <td>0.000118</td>
        </tr>
        <tr>
          <th>Specificity</th>
          <td>0.978723</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 66-67

``specificity`` now appears alongside the built-in metrics.

.. GENERATED FROM PYTHON SOURCE LINES 69-76

Passing extra keyword arguments
================================

If your metric needs extra data at scoring time (e.g. sample-level amounts,
a cost matrix, ...), pass them as keyword arguments to
:meth:`~skore.EstimatorReport.metrics.add`. They will be forwarded to the
metric function when it is computed.

.. GENERATED FROM PYTHON SOURCE LINES 78-96

.. code-block:: Python



    def misclassification_cost(y_true, y_pred, cost_fp, cost_fn):
        """Total cost of misclassifications, weighted by error type."""
        fp = ((y_true == 0) & (y_pred == 1)).sum()
        fn = ((y_true == 1) & (y_pred == 0)).sum()
        return cost_fp * fp + cost_fn * fn


    report.metrics.add(
        misclassification_cost,
        greater_is_better=False,
        cost_fp=1.0,
        cost_fn=10.0,
    )

    report.metrics.summarize().frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>LogisticRegression</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Accuracy</th>
          <td>0.947368</td>
        </tr>
        <tr>
          <th>Precision</th>
          <td>0.984127</td>
        </tr>
        <tr>
          <th>Recall</th>
          <td>0.925373</td>
        </tr>
        <tr>
          <th>ROC AUC</th>
          <td>0.993649</td>
        </tr>
        <tr>
          <th>Log loss</th>
          <td>0.110103</td>
        </tr>
        <tr>
          <th>Brier score</th>
          <td>0.036139</td>
        </tr>
        <tr>
          <th>Fit time (s)</th>
          <td>0.199774</td>
        </tr>
        <tr>
          <th>Predict time (s)</th>
          <td>0.000118</td>
        </tr>
        <tr>
          <th>Specificity</th>
          <td>0.978723</td>
        </tr>
        <tr>
          <th>Misclassification Cost</th>
          <td>51.000000</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 97-103

Adding an sklearn scorer
========================

If you already have a :func:`~sklearn.metrics.make_scorer` object, you can
register it directly. The ``response_method`` and ``greater_is_better``
metadata are extracted from the scorer automatically.

.. GENERATED FROM PYTHON SOURCE LINES 105-112

.. code-block:: Python

    from sklearn.metrics import fbeta_score, make_scorer

    f2_scorer = make_scorer(fbeta_score, beta=2, response_method="predict", pos_label=1)
    report.metrics.add(f2_scorer, name="f2")

    report.metrics.summarize().frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>LogisticRegression</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Accuracy</th>
          <td>0.947368</td>
        </tr>
        <tr>
          <th>Precision</th>
          <td>0.984127</td>
        </tr>
        <tr>
          <th>Recall</th>
          <td>0.925373</td>
        </tr>
        <tr>
          <th>ROC AUC</th>
          <td>0.993649</td>
        </tr>
        <tr>
          <th>Log loss</th>
          <td>0.110103</td>
        </tr>
        <tr>
          <th>Brier score</th>
          <td>0.036139</td>
        </tr>
        <tr>
          <th>Fit time (s)</th>
          <td>0.199774</td>
        </tr>
        <tr>
          <th>Predict time (s)</th>
          <td>0.000118</td>
        </tr>
        <tr>
          <th>Specificity</th>
          <td>0.978723</td>
        </tr>
        <tr>
          <th>Misclassification Cost</th>
          <td>51.000000</td>
        </tr>
        <tr>
          <th>F2</th>
          <td>0.936556</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 113-118

Cherry-picking metrics to display
==================================

Once registered, custom metrics can be selected by name in
:meth:`~skore.EstimatorReport.metrics.summarize`:

.. GENERATED FROM PYTHON SOURCE LINES 120-124

.. code-block:: Python

    report.metrics.summarize(
        metric=["specificity", "f2", "misclassification_cost"],
    ).frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>LogisticRegression</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Specificity</th>
          <td>0.978723</td>
        </tr>
        <tr>
          <th>F2</th>
          <td>0.936556</td>
        </tr>
        <tr>
          <th>Misclassification Cost</th>
          <td>51.000000</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 125-126

Selecting ``data_source="both"`` lets you compare train vs. test in one call:

.. GENERATED FROM PYTHON SOURCE LINES 128-130

.. code-block:: Python

    report.metrics.summarize(metric=["specificity", "f2"], data_source="both").frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>LogisticRegression (train)</th>
          <th>LogisticRegression (test)</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Specificity</th>
          <td>0.933333</td>
          <td>0.978723</td>
        </tr>
        <tr>
          <th>F2</th>
          <td>0.975945</td>
          <td>0.936556</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 131-136

Using a different response method
==================================

By default, callables receive the output of ``estimator.predict(X)``. If your
metric needs probabilities instead, set ``response_method="predict_proba"``.

.. GENERATED FROM PYTHON SOURCE LINES 138-151

.. code-block:: Python

    import numpy as np


    def mean_confidence(y_true, y_proba):
        """Average predicted probability assigned to the true class."""
        return np.where(y_true == 1, y_proba, 1 - y_proba).mean()


    report.metrics.add(
        mean_confidence, response_method="predict_proba", greater_is_better=True
    )

    report.metrics.summarize(metric="mean_confidence").frame()





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>LogisticRegression</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Mean Confidence</th>
          <td>0.931102</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.245 seconds)


.. _sphx_glr_download_auto_examples_model_evaluation_plot_custom_metrics.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_custom_metrics.ipynb <plot_custom_metrics.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_custom_metrics.py <plot_custom_metrics.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_custom_metrics.zip <plot_custom_metrics.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_