.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/technical_details/plot_sklearn_api.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_technical_details_plot_sklearn_api.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_technical_details_plot_sklearn_api.py:


.. _example_sklearn_api:

=====================================================================================
Using skore with scikit-learn compatible estimators
=====================================================================================

This example shows how to use skore with scikit-learn compatible estimators.

Any model that can be used with the scikit-learn API can be used with skore.
Use :func:`~skore.evaluate` to create a report from any estimator that has a
``fit`` and ``predict`` method (or only ``predict`` if already fitted).

.. note::

  When computing the ROC AUC or ROC curve for a classification task, the estimator must
  have a ``predict_proba`` method.

In this example, we showcase a gradient boosting model
(`XGBoost <https://github.com/dmlc/xgboost>`_) and a custom estimator.

Note that this example is not exhaustive; many other scikit-learn compatible models can
be used with skore:

-   More gradient boosting libraries like
    `LightGBM <https://github.com/microsoft/LightGBM>`_, and
    `CatBoost <https://github.com/catboost/catboost>`_,

-   Deep learning frameworks such as `Keras <https://github.com/keras-team/keras>`_
    and `skorch <https://github.com/skorch-dev/skorch>`_ (a wrapper for
    `PyTorch <https://github.com/pytorch/pytorch>`_).

-   Tabular foundation models such as
    `TabICL <https://github.com/soda-inria/tabicl>`_ and
    `TabPFN <https://github.com/PriorLabs/TabPFN>`_,

-   etc.

.. GENERATED FROM PYTHON SOURCE LINES 42-47

Loading a binary classification dataset
=======================================

We generate a synthetic binary classification dataset with only 1,000 samples to keep
the computation time reasonable:

.. GENERATED FROM PYTHON SOURCE LINES 49-54

.. code-block:: Python

    from sklearn.datasets import make_classification

    X, y = make_classification(n_samples=1_000, random_state=42)
    print(f"{X.shape = }")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    X.shape = (1000, 20)




.. GENERATED FROM PYTHON SOURCE LINES 55-61

Gradient-boosted decision trees with XGBoost
============================================

For this binary classification task, we consider a gradient-boosted decision trees
model from a library external to scikit-learn.
One of the most popular is `XGBoost <https://github.com/dmlc/xgboost>`_.

.. GENERATED FROM PYTHON SOURCE LINES 63-71

.. code-block:: Python

    from skore import evaluate
    from xgboost import XGBClassifier

    xgb = XGBClassifier(n_estimators=50, max_depth=3, learning_rate=0.1, random_state=42)

    xgb_report = evaluate(xgb, X, y, splitter=0.2, pos_label=1)
    xgb_report.metrics.summarize().frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>XGBClassifier</th>
        </tr>
        <tr>
          <th>Metric</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>Accuracy</th>
          <td>0.900000</td>
        </tr>
        <tr>
          <th>Precision</th>
          <td>0.989899</td>
        </tr>
        <tr>
          <th>Recall</th>
          <td>0.837607</td>
        </tr>
        <tr>
          <th>ROC AUC</th>
          <td>0.980126</td>
        </tr>
        <tr>
          <th>Log loss</th>
          <td>0.218888</td>
        </tr>
        <tr>
          <th>Brier score</th>
          <td>0.064364</td>
        </tr>
        <tr>
          <th>Fit time (s)</th>
          <td>0.029269</td>
        </tr>
        <tr>
          <th>Predict time (s)</th>
          <td>0.000341</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 72-73

We can easily get the summary of metrics, and also a ROC curve plot for example:

.. GENERATED FROM PYTHON SOURCE LINES 75-77

.. code-block:: Python

    xgb_report.metrics.roc().plot()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    <Figure size 600x750 with 1 Axes>



.. GENERATED FROM PYTHON SOURCE LINES 78-79

We can also inspect our model:

.. GENERATED FROM PYTHON SOURCE LINES 81-83

.. code-block:: Python

    xgb_report.inspection.permutation_importance().frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>data_source</th>
          <th>metric</th>
          <th>feature</th>
          <th>value_mean</th>
          <th>value_std</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #0</td>
          <td>0.001</td>
          <td>0.002236</td>
        </tr>
        <tr>
          <th>1</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #1</td>
          <td>0.003</td>
          <td>0.006708</td>
        </tr>
        <tr>
          <th>2</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #2</td>
          <td>-0.012</td>
          <td>0.005701</td>
        </tr>
        <tr>
          <th>3</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #3</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>4</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #4</td>
          <td>-0.002</td>
          <td>0.002739</td>
        </tr>
        <tr>
          <th>5</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #5</td>
          <td>0.348</td>
          <td>0.044102</td>
        </tr>
        <tr>
          <th>6</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #6</td>
          <td>-0.001</td>
          <td>0.002236</td>
        </tr>
        <tr>
          <th>7</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #7</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>8</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #8</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>9</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #9</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>10</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #10</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>11</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #11</td>
          <td>-0.004</td>
          <td>0.010840</td>
        </tr>
        <tr>
          <th>12</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #12</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>13</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #13</td>
          <td>-0.003</td>
          <td>0.002739</td>
        </tr>
        <tr>
          <th>14</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #14</td>
          <td>0.022</td>
          <td>0.010954</td>
        </tr>
        <tr>
          <th>15</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #15</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>16</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #16</td>
          <td>0.000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>17</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #17</td>
          <td>-0.002</td>
          <td>0.002739</td>
        </tr>
        <tr>
          <th>18</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #18</td>
          <td>-0.006</td>
          <td>0.002236</td>
        </tr>
        <tr>
          <th>19</th>
          <td>test</td>
          <td>accuracy</td>
          <td>Feature #19</td>
          <td>0.001</td>
          <td>0.002236</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 84-86

Custom model
------------

.. GENERATED FROM PYTHON SOURCE LINES 88-91

Let us use a custom estimator inspired from the
`scikit-learn documentation <https://scikit-learn.org/dev/developers/develop.html#rolling-your-own-estimator>`_,
a nearest neighbor classifier:

.. GENERATED FROM PYTHON SOURCE LINES 93-118

.. code-block:: Python

    import numpy as np
    from sklearn.base import BaseEstimator, ClassifierMixin
    from sklearn.metrics import euclidean_distances
    from sklearn.utils.multiclass import unique_labels
    from sklearn.utils.validation import check_is_fitted, validate_data


    class CustomClassifier(ClassifierMixin, BaseEstimator):
        def __init__(self):
            pass

        def fit(self, X, y):
            X, y = validate_data(self, X, y)
            self.classes_ = unique_labels(y)
            self.X_ = X
            self.y_ = y
            return self

        def predict(self, X):
            check_is_fitted(self)
            X = validate_data(self, X, reset=False)
            closest = np.argmin(euclidean_distances(X, self.X_), axis=1)
            return self.y_[closest]









.. GENERATED FROM PYTHON SOURCE LINES 119-123

.. note::

  The estimator above does not have a `predict_proba` method, therefore
  we cannot display its ROC curve as done previously.

.. GENERATED FROM PYTHON SOURCE LINES 125-126

We can now use this model with skore:

.. GENERATED FROM PYTHON SOURCE LINES 128-131

.. code-block:: Python

    custom_report = evaluate(CustomClassifier(), X, y, splitter=0.2, pos_label=1)
    custom_report.metrics.precision()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none



    0.7864077669902912



.. GENERATED FROM PYTHON SOURCE LINES 132-152

Conclusion
==========

This example demonstrates how skore can be used with scikit-learn compatible estimators.
This allows practitioners to use consistent reporting and visualization tools across different estimators.

.. seealso::

  For a practical example of using language models within scikit-learn pipelines,
  see :ref:`example_use_case_employee_salaries` which demonstrates how to use
  skrub's :class:`~skrub.TextEncoder` (a language model-based encoder) in a
  scikit-learn pipeline for feature engineering.

.. seealso::

  For an example of wrapping Large Language Models (LLMs) to be compatible with
  scikit-learn APIs, see the tutorial on `Quantifying LLMs Uncertainty with Conformal
  Predictions <https://medium.com/capgemini-invent-lab/quantifying-llms-uncertainty-with-conformal-predictions-567870e63e00>`_.
  The article demonstrates how to wrap models like Mistral-7B-Instruct in a
  scikit-learn-compatible interface.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.294 seconds)


.. _sphx_glr_download_auto_examples_technical_details_plot_sklearn_api.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_sklearn_api.ipynb <plot_sklearn_api.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_sklearn_api.py <plot_sklearn_api.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_sklearn_api.zip <plot_sklearn_api.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_