{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# `EstimatorReport.metrics.add`: Adapt skore to your use-case\n\nBy default, :meth:`~skore.EstimatorReport.metrics.summarize` reports a curated\nset of metrics for your ML task. In practice you often need domain-specific\nscores: a business cost function, a custom fairness measure, an F-beta with a\nparticular beta, etc.\n\nThis example walks through how to register such metrics with\n:meth:`~skore.EstimatorReport.metrics.add` so they are computed and displayed\nalongside the built-in ones.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setting up a classification problem\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import skore\nfrom sklearn.datasets import load_breast_cancer\nfrom sklearn.linear_model import LogisticRegression\n\nX, y = load_breast_cancer(return_X_y=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We create an :class:`~skore.EstimatorReport` through :func:`~skore.evaluate`\nusing a simple train/test split. ``pos_label=1`` marks the *malignant* class\nas the positive class.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report = skore.evaluate(\n    LogisticRegression(max_iter=10_000), X, y, pos_label=1, splitter=0.2\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's look at the default metrics:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize().frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Adding a plain callable\n\nAny function with the signature ``(y_true, y_pred, **kwargs) -> float`` can be\nregistered with :meth:`~skore.EstimatorReport.metrics.add`. The function name\nis used as the metric name by default.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def specificity(y_true, y_pred):\n    \"\"\"Proportion of true negatives among actual negatives.\"\"\"\n    tn = ((y_true == 0) & (y_pred == 0)).sum()\n    fp = ((y_true == 0) & (y_pred == 1)).sum()\n    return tn / (tn + fp)\n\n\nreport.metrics.add(specificity, greater_is_better=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize().frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "``specificity`` now appears alongside the built-in metrics.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Passing extra keyword arguments\n\nIf your metric needs extra data at scoring time (e.g. sample-level amounts,\na cost matrix, ...), pass them as keyword arguments to\n:meth:`~skore.EstimatorReport.metrics.add`. They will be forwarded to the\nmetric function when it is computed.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def misclassification_cost(y_true, y_pred, cost_fp, cost_fn):\n    \"\"\"Total cost of misclassifications, weighted by error type.\"\"\"\n    fp = ((y_true == 0) & (y_pred == 1)).sum()\n    fn = ((y_true == 1) & (y_pred == 0)).sum()\n    return cost_fp * fp + cost_fn * fn\n\n\nreport.metrics.add(\n    misclassification_cost,\n    greater_is_better=False,\n    cost_fp=1.0,\n    cost_fn=10.0,\n)\n\nreport.metrics.summarize().frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Adding an sklearn scorer\n\nIf you already have a :func:`~sklearn.metrics.make_scorer` object, you can\nregister it directly. The ``response_method`` and ``greater_is_better``\nmetadata are extracted from the scorer automatically.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import fbeta_score, make_scorer\n\nf2_scorer = make_scorer(fbeta_score, beta=2, response_method=\"predict\", pos_label=1)\nreport.metrics.add(f2_scorer, name=\"f2\")\n\nreport.metrics.summarize().frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Cherry-picking metrics to display\n\nOnce registered, custom metrics can be selected by name in\n:meth:`~skore.EstimatorReport.metrics.summarize`:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize(\n    metric=[\"specificity\", \"f2\", \"misclassification_cost\"],\n).frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Selecting ``data_source=\"both\"`` lets you compare train vs. test in one call:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize(metric=[\"specificity\", \"f2\"], data_source=\"both\").frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Using a different response method\n\nBy default, callables receive the output of ``estimator.predict(X)``. If your\nmetric needs probabilities instead, set ``response_method=\"predict_proba\"``.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\n\n\ndef mean_confidence(y_true, y_proba):\n    \"\"\"Average predicted probability assigned to the true class.\"\"\"\n    return np.where(y_true == 1, y_proba, 1 - y_proba).mean()\n\n\nreport.metrics.add(\n    mean_confidence, response_method=\"predict_proba\", greater_is_better=True\n)\n\nreport.metrics.summarize(metric=\"mean_confidence\").frame()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}