.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/model_evaluation/plot_estimator_report.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_model_evaluation_plot_estimator_report.py: .. _example_estimator_report: =============================================================== `EstimatorReport`: Get insights from any scikit-learn estimator =============================================================== This example shows how the :class:`skore.EstimatorReport` class can be used to quickly get insights from any scikit-learn estimator. .. GENERATED FROM PYTHON SOURCE LINES 13-25 Loading our dataset and defining our estimator ============================================== First, we load a dataset from skrub. Our goal is to predict whether an online shopping basket is fraudulent, so that the payment can be reviewed before money leaves the account. Baskets and products live in two tables, so we aggregate product-level information (including the cash amount of the basket) into one feature matrix with pandas. Using a skrub DataOp to keep those joins inside the estimator (and replay them on unseen data) is shown in :ref:`example_data_processing`. .. GENERATED FROM PYTHON SOURCE LINES 27-59 .. code-block:: Python from skrub.datasets import fetch_credit_fraud dataset = fetch_credit_fraud(split="train") baskets = dataset.baskets products = dataset.products basket_features = ( products.groupby("basket_ID") .agg( basket_amount=("cash_price", "sum"), n_items=("cash_price", "count"), mean_item_price=("cash_price", "mean"), max_item_price=("cash_price", "max"), n_makes=("make", "nunique"), n_item_types=("item", "nunique"), ) .reset_index() ) top_product = ( products.sort_values("cash_price", ascending=False) .groupby("basket_ID", as_index=False) .first()[["basket_ID", "make", "item"]] .rename(columns={"make": "top_make", "item": "top_item"}) ) df = ( baskets.merge(basket_features, left_on="ID", right_on="basket_ID") .merge(top_product, on="basket_ID") .drop(columns=["ID", "basket_ID"]) ) y = df.pop("fraud_flag") .. GENERATED FROM PYTHON SOURCE LINES 60-64 .. code-block:: Python from skrub import TableReport TableReport(df) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 65-67 .. code-block:: Python TableReport(y.to_frame()) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 68-75 Looking at the distributions of the target, we observe that this classification task is quite imbalanced. This means that we have to be careful when selecting a set of statistical metrics to evaluate the classification performance of our predictive model. In addition, we see that the class labels are specified by an integer 0 or 1. For our application, the label of interest is ``1`` (fraudulent). .. GENERATED FROM PYTHON SOURCE LINES 75-77 .. code-block:: Python pos_label, neg_label = 1, 0 .. GENERATED FROM PYTHON SOURCE LINES 78-85 Let's create a predictive model. Thankfully, `skrub` provides a convenient function (:func:`skrub.tabular_pipeline`) when it comes to getting strong baseline predictive models with a single line of code. As its feature engineering is generic, it does not provide some handcrafted and tailored feature engineering but still provides a good starting point. So let's create a classifier for our task. .. GENERATED FROM PYTHON SOURCE LINES 85-90 .. code-block:: Python from skrub import tabular_pipeline estimator = tabular_pipeline("classifier") estimator .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingclassifier',
                     HistGradientBoostingClassifier())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 91-98 Introducing the :class:`EstimatorReport` ======================================== Let's gather some insights from our predictive model. We can use :func:`skore.evaluate` for this: the function will perform a train-test split and create a :class:`~skore.EstimatorReport` containing the model fitted on the training data, ready to investigate. .. GENERATED FROM PYTHON SOURCE LINES 98-105 .. code-block:: Python from skore import evaluate # Reserve 20% of the data for the test set report = evaluate(estimator, X=df, y=y, pos_label=pos_label, splitter=0.2) report .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingclassifier',
                     HistGradientBoostingClassifier())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 106-109 Once the report is created, we get some information regarding the available tools allowing us to get some insights on our model by calling the :meth:`~skore.EstimatorReport.help` method. .. GENERATED FROM PYTHON SOURCE LINES 110-112 .. code-block:: Python report.help() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 113-114 Be aware that we can access the help for each individual sub-accessor. For instance: .. GENERATED FROM PYTHON SOURCE LINES 115-117 .. code-block:: Python report.metrics.help() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 118-124 Measuring model performance =========================== Let's have a first look at the statistical performance of our model. skore knows that we are doing classification, and can give us an array of classic ML metrics, all at once, with :meth:`~skore.EstimatorReport.metrics.summarize`: .. GENERATED FROM PYTHON SOURCE LINES 125-132 .. code-block:: Python import time start = time.time() metric_report = report.metrics.summarize().frame() end = time.time() metric_report .. rst-class:: sphx-glr-script-out .. code-block:: none metric accuracy 0.987183 precision 0.583333 recall 0.044025 roc_auc 0.850833 log_loss 0.055880 brier_score 0.011915 fit_time 2.756917 predict_time 0.337309 Name: HistGradientBoostingClassifier, dtype: float64 .. GENERATED FROM PYTHON SOURCE LINES 133-135 .. code-block:: Python print(f"Time taken to compute the metrics: {end - start:.2f} seconds") .. rst-class:: sphx-glr-script-out .. code-block:: none Time taken to compute the metrics: 0.00 seconds .. GENERATED FROM PYTHON SOURCE LINES 136-138 Since the output is a pandas dataframe, we can also use the plotting interface of pandas. .. GENERATED FROM PYTHON SOURCE LINES 139-142 .. code-block:: Python ax = metric_report.plot.barh() _ = ax.set_title("Metrics report") .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_001.png :alt: Metrics report :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 143-149 An interesting feature of the :class:`skore.EstimatorReport` is its caching mechanism. Indeed, when we have a large enough dataset, computing the predictions for a model can be expensive. To amortize this cost, the report will cache the predictions when it is first created; this way, calculations that need the model predictions can get them from the cache and save a lot of time. This is why the metrics computation above is so fast. .. GENERATED FROM PYTHON SOURCE LINES 152-154 When the model is fitted or the predictions are computed, we additionally store the time the operation took: .. GENERATED FROM PYTHON SOURCE LINES 155-157 .. code-block:: Python report.metrics.timings() .. rst-class:: sphx-glr-script-out .. code-block:: none {'fit_time': 2.756917477000002, 'predict_time_train': 1.2270488629999932, 'predict_time_test': 0.33730852500002584} .. GENERATED FROM PYTHON SOURCE LINES 158-160 By default, the metrics are computed on the test set only, but we can also compute them on the train set: .. GENERATED FROM PYTHON SOURCE LINES 161-164 .. code-block:: Python report.metrics.log_loss(data_source="train") .. rst-class:: sphx-glr-script-out .. code-block:: none 0.0471055344082289 .. GENERATED FROM PYTHON SOURCE LINES 165-171 Defining custom metrics ======================= skore can compute user-defined metrics as well. It accepts metrics in the form of scikit-learn scorers, i.e. functions taking `estimator`, `X` and `y` (and optional keyword arguments). Let's take a look at an example. .. GENERATED FROM PYTHON SOURCE LINES 172-206 .. code-block:: Python def operational_decision_gain(y_true, y_pred, *, amount): """The monetary gain we obtain depending on our predictions. May be negative, in which case our predictions actually *cost* us money. """ mask_true_positive = (y_true == pos_label) & (y_pred == pos_label) mask_true_negative = (y_true == neg_label) & (y_pred == neg_label) mask_false_positive = (y_true == neg_label) & (y_pred == pos_label) mask_false_negative = (y_true == pos_label) & (y_pred == neg_label) REVIEW_COST = -20 REPUTATION_COST = -30 MARGIN = 0.02 # Fraud correctly flagged: we pay the review costs, but do not incur # more cost fraudulent_refuse = mask_true_positive.sum() * REVIEW_COST # Fraud missed: the payment goes through and we lose the full basket amount fraudulent_accept = -amount[mask_false_negative].sum() # Legitimate basket wrongly flagged: we pay the review costs, but we also annoy # the customer and risk losing them, so it is penalized compared to a correct # refusal legitimate_refuse = mask_false_positive.sum() * (REVIEW_COST + REPUTATION_COST) # Legitimate basket correctly accepted: we earn a margin on the sale legitimate_accept = (amount[mask_true_negative] * MARGIN).sum() return fraudulent_refuse + fraudulent_accept + legitimate_refuse + legitimate_accept .. GENERATED FROM PYTHON SOURCE LINES 207-211 In our example use case, each classification decision has a different monetary gain. The function above models this by translating the confusion matrix into a gain (payoff) matrix that depends on the basket cash value. Let's test adding this metric to our report. .. GENERATED FROM PYTHON SOURCE LINES 212-220 .. code-block:: Python from sklearn.metrics import make_scorer amount = report.X_test["basket_amount"] # We use `make_scorer` to convert the metric to the right format (a function # that takes `estimator`, `X`, `y`) report.metrics.add(metric=make_scorer(operational_decision_gain, amount=amount)) .. GENERATED FROM PYTHON SOURCE LINES 221-224 Our custom metric is now registered in the report, and will be shown in the summary. In fact, since the underlying metric function takes `y_pred` as input, skore can use the cached predictions again to speed up the computation. .. GENERATED FROM PYTHON SOURCE LINES 225-229 .. code-block:: Python # The metric name is derived from the function name unless it is explicitly given report.metrics.summarize().frame() .. rst-class:: sphx-glr-script-out .. code-block:: none metric operational_decision_gain 72169.700000 accuracy 0.987183 precision 0.583333 recall 0.044025 roc_auc 0.850833 log_loss 0.055880 brier_score 0.011915 fit_time 2.756917 predict_time 0.337309 Name: HistGradientBoostingClassifier, dtype: float64 .. GENERATED FROM PYTHON SOURCE LINES 230-237 Effortless one-liner plotting ============================= The :class:`skore.EstimatorReport` class also implements a number of the most common data science plots. As for the metrics, we only provide the meaningful set of plots for the provided estimator. .. GENERATED FROM PYTHON SOURCE LINES 238-240 .. code-block:: Python report.metrics.help() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 241-242 Let's plot the ROC curve for our binary classification task. .. GENERATED FROM PYTHON SOURCE LINES 243-246 .. code-block:: Python display = report.metrics.roc() display.plot() .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_002.png :alt: ROC Curve for HistGradientBoostingClassifier Positive label: 1 Data source: Test set :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none
.. GENERATED FROM PYTHON SOURCE LINES 247-251 The plot functionality is built upon the scikit-learn Display objects. We return those Display objects (slightly modified to improve the UI) in case we want to tweak some of the plot properties. We can have a quick look at the available attributes and methods by calling the ``help`` method. .. GENERATED FROM PYTHON SOURCE LINES 252-254 .. code-block:: Python display.help() .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 255-259 .. code-block:: Python fig = display.plot() fig.axes[0].set_title("Example of a ROC curve") fig .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_003.png :alt: ROC Curve for HistGradientBoostingClassifier Positive label: 1 Data source: Test set, Example of a ROC curve :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_003.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none
.. GENERATED FROM PYTHON SOURCE LINES 260-262 Similarly to the metrics, the cache allows us to avoid recomputing the model predictions, which speeds up the display generation. .. GENERATED FROM PYTHON SOURCE LINES 263-269 .. code-block:: Python start = time.time() display = report.metrics.roc() _ = display.plot() end = time.time() print(f"Time taken to compute the ROC curve: {end - start:.2f} seconds") .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_004.png :alt: ROC Curve for HistGradientBoostingClassifier Positive label: 1 Data source: Test set :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_004.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Time taken to compute the ROC curve: 0.11 seconds .. GENERATED FROM PYTHON SOURCE LINES 270-272 You can learn more about the cache system in the corresponding example: :ref:`example_cache_mechanism`. .. GENERATED FROM PYTHON SOURCE LINES 274-279 Visualizing the confusion matrix ================================ Another useful visualization for classification tasks is the confusion matrix, which shows the counts of correct and incorrect predictions for each class. .. GENERATED FROM PYTHON SOURCE LINES 281-282 Let's start with a basic confusion matrix: .. GENERATED FROM PYTHON SOURCE LINES 282-285 .. code-block:: Python cm_display = report.metrics.confusion_matrix() cm_display.plot() .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_005.png :alt: Confusion Matrix Data source: Test set :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_005.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none
.. GENERATED FROM PYTHON SOURCE LINES 286-294 In binary classification, a confusion matrix depends on the decision threshold used to convert predicted probabilities into class labels. By default, skore uses a threshold of 0.5, but confusion matrices are actually computed at every threshold internally. To visualize the confusion matrix at a different threshold, use the ``threshold_value`` parameter. For example, a threshold of 0.1 will classify more samples as positive: .. GENERATED FROM PYTHON SOURCE LINES 294-296 .. code-block:: Python cm_display.plot(threshold_value=0.1) .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_006.png :alt: Confusion Matrix Decision threshold: 0.10 Positive label: 1 Data source: Test set :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_006.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none
.. GENERATED FROM PYTHON SOURCE LINES 297-299 We can normalize the confusion matrix to get percentages instead of raw counts. Here we normalize by true labels (rows): .. GENERATED FROM PYTHON SOURCE LINES 299-301 .. code-block:: Python cm_display.plot(normalize="true") .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_007.png :alt: Confusion Matrix Data source: Test set :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_007.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none
.. GENERATED FROM PYTHON SOURCE LINES 302-304 More plotting options are available via ``heatmap_kwargs``, which are passed to seaborn's heatmap. For example, we can customize the colormap and number format: .. GENERATED FROM PYTHON SOURCE LINES 304-307 .. code-block:: Python cm_display.set_style(heatmap_kwargs={"cmap": "Greens", "fmt": ".2e"}) cm_display.plot() .. image-sg:: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_008.png :alt: Confusion Matrix Data source: Test set :srcset: /auto_examples/model_evaluation/images/sphx_glr_plot_estimator_report_008.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none
.. GENERATED FROM PYTHON SOURCE LINES 308-310 Finally, the confusion matrix can also be exported as a pandas DataFrame for further analysis: .. GENERATED FROM PYTHON SOURCE LINES 310-312 .. code-block:: Python cm_display.frame() .. raw:: html
true_label predicted_label value
0 0 0 12085
1 0 1 5
2 1 0 152
3 1 1 7


.. GENERATED FROM PYTHON SOURCE LINES 313-317 .. seealso:: For using the :class:`~skore.EstimatorReport` to inspect your models, see :ref:`example_feature_importance`. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 10.800 seconds) .. _sphx_glr_download_auto_examples_model_evaluation_plot_estimator_report.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_estimator_report.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_estimator_report.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_estimator_report.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_