.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/getting_started/plot_skore_getting_started.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_getting_started_plot_skore_getting_started.py: .. _example_skore_getting_started: ====================== Skore: getting started ====================== .. GENERATED FROM PYTHON SOURCE LINES 10-27 This getting started guide illustrates how to use skore and why: #. Get assistance when developing your machine learning projects to avoid common pitfalls and follow recommended practices. * :class:`skore.EstimatorReport`: get an insightful report on your estimator, for evaluation and inspection * :class:`skore.CrossValidationReport`: get an insightful report on your cross-validation results * :class:`skore.ComparisonReport`: benchmark your skore estimator reports * :func:`skore.train_test_split`: get diagnostics when splitting your data #. Track your machine learning results using skore's :class:`~skore.Project` (for storage). .. GENERATED FROM PYTHON SOURCE LINES 29-35 Machine learning evaluation and diagnostics =========================================== Skore implements new tools or wraps some key scikit-learn class / functions to automatically provide insights and diagnostics when using them, as a way to facilitate good practices and avoid common pitfalls. .. GENERATED FROM PYTHON SOURCE LINES 37-45 Model evaluation with skore ^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to assist its users when programming, skore has implemented a :class:`skore.EstimatorReport` class. Let us create a challenging synthetic binary classification dataset and get the estimator report for a :class:`~sklearn.ensemble.RandomForestClassifier`: .. GENERATED FROM PYTHON SOURCE LINES 47-68 .. code-block:: Python from sklearn.datasets import make_classification from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split from skore import EstimatorReport X, y = make_classification( n_samples=10_000, n_classes=3, class_sep=0.3, n_clusters_per_class=1, random_state=42, ) X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0) rf = RandomForestClassifier(random_state=0) rf_report = EstimatorReport( rf, X_train=X_train, X_test=X_test, y_train=y_train, y_test=y_test ) .. GENERATED FROM PYTHON SOURCE LINES 69-71 Now, we can display the helper to see all the insights that are available to us (skore detected that we are doing multiclass classification): .. GENERATED FROM PYTHON SOURCE LINES 73-75 .. code-block:: Python rf_report.help() .. rst-class:: sphx-glr-script-out .. code-block:: none ╭───────────────── Tools to diagnose estimator RandomForestClassifier ─────────────────╮ │ EstimatorReport │ │ ├── .metrics │ │ │ ├── .accuracy(...) (↗︎) - Compute the accuracy score. │ │ │ ├── .confusion_matrix(...) - Plot the confusion matrix. │ │ │ ├── .log_loss(...) (↘︎) - Compute the log loss. │ │ │ ├── .precision(...) (↗︎) - Compute the precision score. │ │ │ ├── .precision_recall(...) - Plot the precision-recall curve. │ │ │ ├── .recall(...) (↗︎) - Compute the recall score. │ │ │ ├── .roc(...) - Plot the ROC curve. │ │ │ ├── .roc_auc(...) (↗︎) - Compute the ROC AUC score. │ │ │ ├── .timings(...) - Get all measured processing times related │ │ │ │ to the estimator. │ │ │ ├── .custom_metric(...) - Compute a custom metric. │ │ │ └── .summarize(...) - Report a set of metrics for our estimator. │ │ ├── .feature_importance │ │ │ ├── .mean_decrease_impurity(...) - Retrieve the mean decrease impurity (MDI) │ │ │ │ of a tree-based model. │ │ │ └── .permutation(...) - Report the permutation feature importance. │ │ ├── .data │ │ │ └── .analyze(...) - Plot dataset statistics. │ │ ├── .cache_predictions(...) - Cache estimator's predictions. │ │ ├── .clear_cache(...) - Clear the cache. │ │ ├── .get_predictions(...) - Get estimator's predictions. │ │ └── Attributes │ │ ├── .X_test - Testing data │ │ ├── .X_train - Training data │ │ ├── .y_test - Testing target │ │ ├── .y_train - Training target │ │ ├── .estimator - Estimator to make the report from │ │ ├── .estimator_ - The cloned or copied estimator │ │ ├── .estimator_name_ - The name of the estimator │ │ ├── .fit - Whether to fit the estimator on the │ │ │ training data │ │ ├── .fit_time_ - The time taken to fit the estimator, in │ │ │ seconds │ │ ├── .ml_task - No description available │ │ └── .pos_label - For binary classification, the positive │ │ class │ │ │ │ │ │ Legend: │ │ (↗︎) higher is better (↘︎) lower is better │ ╰──────────────────────────────────────────────────────────────────────────────────────╯ .. GENERATED FROM PYTHON SOURCE LINES 76-83 .. note:: This helper is great because: - it enables users to get a glimpse at the API of the different available accessors without having to look up the online documentation, - it provides methodological guidance: for example, we easily provide several metrics as a way to encourage users looking into them. .. GENERATED FROM PYTHON SOURCE LINES 85-88 We can evaluate our model using the :meth:`~skore.EstimatorReport.metrics` accessor. In particular, we can get the report metrics that is computed for us (including the fit and prediction times): .. GENERATED FROM PYTHON SOURCE LINES 90-92 .. code-block:: Python rf_report.metrics.summarize(indicator_favorability=True).frame() .. raw:: html
RandomForestClassifier Favorability
Metric Label / Average
Precision 0 0.878468 (↗︎)
1 0.696411 (↗︎)
2 0.730337 (↗︎)
Recall 0 0.768786 (↗︎)
1 0.850711 (↗︎)
2 0.657396 (↗︎)
ROC AUC 0 0.945966 (↗︎)
1 0.906700 (↗︎)
2 0.868317 (↗︎)
Log loss 0.550382 (↘︎)
Fit time (s) 3.443031 (↘︎)
Predict time (s) 0.032770 (↘︎)


.. GENERATED FROM PYTHON SOURCE LINES 93-95 For inspection, we can also retrieve the predictions, on the train set for example (here we display only the first 10 predictions for conciseness purposes): .. GENERATED FROM PYTHON SOURCE LINES 97-99 .. code-block:: Python rf_report.get_predictions(data_source="train")[0:10] .. rst-class:: sphx-glr-script-out .. code-block:: none array([2, 1, 2, 2, 2, 1, 0, 0, 0, 0]) .. GENERATED FROM PYTHON SOURCE LINES 100-101 We can also plot the ROC curve that is generated for us: .. GENERATED FROM PYTHON SOURCE LINES 103-106 .. code-block:: Python roc_plot = rf_report.metrics.roc() roc_plot.plot() .. image-sg:: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_001.png :alt: ROC Curve for RandomForestClassifier :srcset: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 107-110 Furthermore, we can inspect our model using the :meth:`~skore.EstimatorReport.feature_importance` accessor. In particular, we can inspect the model using the permutation feature importance: .. GENERATED FROM PYTHON SOURCE LINES 112-117 .. code-block:: Python import matplotlib.pyplot as plt rf_report.feature_importance.permutation(seed=0).T.boxplot(vert=False) plt.tight_layout() .. image-sg:: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_002.png :alt: plot skore getting started :srcset: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 118-125 .. seealso:: For more information about the motivation and usage of :class:`skore.EstimatorReport`, see the following use cases: - :ref:`example_estimator_report` for model evaluation, - :ref:`example_feature_importance` for model inspection. .. GENERATED FROM PYTHON SOURCE LINES 127-132 Cross-validation with skore ^^^^^^^^^^^^^^^^^^^^^^^^^^^ skore has also (re-)implemented a :class:`skore.CrossValidationReport` class that contains several :class:`skore.EstimatorReport`, one for each split. .. GENERATED FROM PYTHON SOURCE LINES 134-138 .. code-block:: Python from skore import CrossValidationReport cv_report = CrossValidationReport(rf, X, y, splitter=5) .. GENERATED FROM PYTHON SOURCE LINES 139-140 We display the cross-validation report helper: .. GENERATED FROM PYTHON SOURCE LINES 142-144 .. code-block:: Python cv_report.help() .. rst-class:: sphx-glr-script-out .. code-block:: none ╭───────────────── Tools to diagnose estimator RandomForestClassifier ─────────────────╮ │ CrossValidationReport │ │ ├── .metrics │ │ │ ├── .accuracy(...) (↗︎) - Compute the accuracy score. │ │ │ ├── .log_loss(...) (↘︎) - Compute the log loss. │ │ │ ├── .precision(...) (↗︎) - Compute the precision score. │ │ │ ├── .precision_recall(...) - Plot the precision-recall curve. │ │ │ ├── .recall(...) (↗︎) - Compute the recall score. │ │ │ ├── .roc(...) - Plot the ROC curve. │ │ │ ├── .roc_auc(...) (↗︎) - Compute the ROC AUC score. │ │ │ ├── .timings(...) - Get all measured processing times related │ │ │ │ to the estimator. │ │ │ ├── .custom_metric(...) - Compute a custom metric. │ │ │ └── .summarize(...) - Report a set of metrics for our estimator. │ │ ├── .cache_predictions(...) - Cache the predictions for sub-estimators │ │ │ reports. │ │ ├── .clear_cache(...) - Clear the cache. │ │ ├── .get_predictions(...) - Get estimator's predictions. │ │ └── Attributes │ │ ├── .X - The data to fit │ │ ├── .y - The target variable to try to predict in │ │ │ the case of supervised learning │ │ ├── .estimator - Estimator to make the cross-validation │ │ │ report from │ │ ├── .estimator_ - The cloned or copied estimator │ │ ├── .estimator_name_ - The name of the estimator │ │ ├── .estimator_reports_ - The estimator reports for each split │ │ ├── .ml_task - No description available │ │ ├── .n_jobs - Number of jobs to run in parallel │ │ ├── .pos_label - For binary classification, the positive │ │ │ class │ │ ├── .split_indices - No description available │ │ └── .splitter - Determines the cross-validation splitting │ │ strategy │ │ │ │ │ │ Legend: │ │ (↗︎) higher is better (↘︎) lower is better │ ╰──────────────────────────────────────────────────────────────────────────────────────╯ .. GENERATED FROM PYTHON SOURCE LINES 145-146 We display the mean and standard deviation for each metric: .. GENERATED FROM PYTHON SOURCE LINES 148-150 .. code-block:: Python cv_report.metrics.summarize().frame() .. raw:: html
RandomForestClassifier
mean std
Metric Label / Average
Precision 0 0.873490 0.020273
1 0.682222 0.014468
2 0.754449 0.008131
Recall 0 0.773915 0.020986
1 0.860532 0.013944
2 0.642843 0.022776
ROC AUC 0 0.947894 0.006489
1 0.900345 0.005975
2 0.868856 0.006914
Log loss 0.599168 0.045947
Fit time (s) 3.755557 0.043977
Predict time (s) 0.028120 0.000410


.. GENERATED FROM PYTHON SOURCE LINES 151-152 or by individual split: .. GENERATED FROM PYTHON SOURCE LINES 154-156 .. code-block:: Python cv_report.metrics.summarize(aggregate=None).frame() .. raw:: html
RandomForestClassifier
Split #0 Split #1 Split #2 Split #3 Split #4
Metric Label / Average
Precision 0 0.844517 0.885191 0.885000 0.892473 0.860269
1 0.682039 0.694377 0.670901 0.665148 0.698647
2 0.748673 0.762478 0.764045 0.750000 0.747049
Recall 0 0.771300 0.795217 0.793722 0.745509 0.763827
1 0.845113 0.854135 0.873684 0.876877 0.852853
2 0.635135 0.665165 0.612613 0.635135 0.666165
ROC AUC 0 0.937561 0.948822 0.955567 0.949205 0.948317
1 0.901664 0.904075 0.900480 0.890213 0.905291
2 0.871745 0.876269 0.865363 0.858650 0.872252
Log loss 0.630675 0.557781 0.544754 0.650172 0.612455
Fit time (s) 3.786378 3.784523 3.764173 3.763388 3.679325
Predict time (s) 0.028784 0.028101 0.028131 0.027889 0.027697


.. GENERATED FROM PYTHON SOURCE LINES 157-158 We display the ROC curves for each split: .. GENERATED FROM PYTHON SOURCE LINES 160-163 .. code-block:: Python roc_plot_cv = cv_report.metrics.roc() roc_plot_cv.plot() .. image-sg:: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_003.png :alt: ROC Curve for RandomForestClassifier :srcset: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 164-166 We can retrieve the estimator report of a specific split to investigate further, for example getting the report metrics for the first split only: .. GENERATED FROM PYTHON SOURCE LINES 168-170 .. code-block:: Python cv_report.estimator_reports_[0].metrics.summarize().frame() .. raw:: html
RandomForestClassifier
Metric Label / Average
Precision 0 0.844517
1 0.682039
2 0.748673
Recall 0 0.771300
1 0.845113
2 0.635135
ROC AUC 0 0.937561
1 0.901664
2 0.871745
Log loss 0.630675
Fit time (s) 3.786378
Predict time (s) 0.028784


.. GENERATED FROM PYTHON SOURCE LINES 171-175 .. seealso:: For more information about the motivation and usage of :class:`skore.CrossValidationReport`, see :ref:`example_use_case_employee_salaries`. .. GENERATED FROM PYTHON SOURCE LINES 177-185 Comparing estimator reports ^^^^^^^^^^^^^^^^^^^^^^^^^^^ :class:`skore.ComparisonReport` enables users to compare several estimator reports (corresponding to several estimators) on a same test set, as in a benchmark of estimators. Apart from the previous ``rf_report``, let us define another estimator report: .. GENERATED FROM PYTHON SOURCE LINES 187-194 .. code-block:: Python from sklearn.ensemble import GradientBoostingClassifier gb = GradientBoostingClassifier(random_state=0) gb_report = EstimatorReport( gb, X_train=X_train, X_test=X_test, y_train=y_train, y_test=y_test, pos_label=1 ) .. GENERATED FROM PYTHON SOURCE LINES 195-197 We can conveniently compare our two estimator reports, that were applied to the exact same test set: .. GENERATED FROM PYTHON SOURCE LINES 199-203 .. code-block:: Python from skore import ComparisonReport comparator = ComparisonReport(reports=[rf_report, gb_report]) .. GENERATED FROM PYTHON SOURCE LINES 204-206 As for the :class:`~skore.EstimatorReport` and the :class:`~skore.CrossValidationReport`, we have a helper: .. GENERATED FROM PYTHON SOURCE LINES 208-210 .. code-block:: Python comparator.help() .. rst-class:: sphx-glr-script-out .. code-block:: none ╭──────────────────────────── Tools to compare estimators ─────────────────────────────╮ │ ComparisonReport │ │ ├── .metrics │ │ │ ├── .accuracy(...) (↗︎) - Compute the accuracy score. │ │ │ ├── .log_loss(...) (↘︎) - Compute the log loss. │ │ │ ├── .precision(...) (↗︎) - Compute the precision score. │ │ │ ├── .precision_recall(...) - Plot the precision-recall curve. │ │ │ ├── .recall(...) (↗︎) - Compute the recall score. │ │ │ ├── .roc(...) - Plot the ROC curve. │ │ │ ├── .roc_auc(...) (↗︎) - Compute the ROC AUC score. │ │ │ ├── .timings(...) - Get all measured processing times related │ │ │ │ to the different estimators. │ │ │ ├── .custom_metric(...) - Compute a custom metric. │ │ │ └── .summarize(...) - Report a set of metrics for the estimators. │ │ ├── .cache_predictions(...) - Cache the predictions for sub-estimators │ │ │ reports. │ │ ├── .clear_cache(...) - Clear the cache. │ │ ├── .get_predictions(...) - Get predictions from the underlying │ │ │ reports. │ │ └── Attributes │ │ ├── .n_jobs - Number of jobs to run in parallel │ │ ├── .pos_label - No description available │ │ └── .reports_ - The compared reports │ │ │ │ │ │ Legend: │ │ (↗︎) higher is better (↘︎) lower is better │ ╰──────────────────────────────────────────────────────────────────────────────────────╯ .. GENERATED FROM PYTHON SOURCE LINES 211-212 Let us display the result of our benchmark: .. GENERATED FROM PYTHON SOURCE LINES 214-216 .. code-block:: Python comparator.metrics.summarize(indicator_favorability=True).frame() .. raw:: html
Estimator RandomForestClassifier GradientBoostingClassifier Favorability
Metric Label / Average
Precision 0 0.878468 0.901370 (↗︎)
1 0.696411 0.707150 (↗︎)
2 0.730337 0.723632 (↗︎)
Recall 0 0.768786 0.760694 (↗︎)
1 0.850711 0.855450 (↗︎)
2 0.657396 0.685209 (↗︎)
ROC AUC 0 0.945966 0.946672 (↗︎)
1 0.906700 0.907682 (↗︎)
2 0.868317 0.869390 (↗︎)
Log loss 0.550382 0.553147 (↘︎)
Fit time (s) 3.443031 18.228429 (↘︎)
Predict time (s) 0.032770 0.010278 (↘︎)


.. GENERATED FROM PYTHON SOURCE LINES 217-218 Thus, we easily have the result of our benchmark for several recommended metrics. .. GENERATED FROM PYTHON SOURCE LINES 220-222 Moreover, we can display the ROC curve for the two estimator reports we want to compare, by superimposing them on the same figure: .. GENERATED FROM PYTHON SOURCE LINES 224-226 .. code-block:: Python comparator.metrics.roc().plot() .. image-sg:: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_004.png :alt: ROC Curve :srcset: /auto_examples/getting_started/images/sphx_glr_plot_skore_getting_started_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 227-234 Train-test split with skore ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Skore has implemented a :func:`skore.train_test_split` function that wraps scikit-learn's :func:`sklearn.model_selection.train_test_split`. Let us load a dataset containing some time series data: .. GENERATED FROM PYTHON SOURCE LINES 236-246 .. code-block:: Python import pandas as pd from skrub.datasets import fetch_employee_salaries dataset_employee = fetch_employee_salaries() X_employee, y_employee = dataset_employee.X, dataset_employee.y X_employee["date_first_hired"] = pd.to_datetime( X_employee["date_first_hired"], format="%m/%d/%Y" ) X_employee.head(2) .. rst-class:: sphx-glr-script-out .. code-block:: none Downloading 'employee_salaries' from https://github.com/skrub-data/skrub-data-files/raw/refs/heads/main/employee_salaries.zip (attempt 1/3) .. raw:: html
gender department department_name division assignment_category employee_position_title date_first_hired year_first_hired
0 F POL Department of Police MSB Information Mgmt and Tech Division Records... Fulltime-Regular Office Services Coordinator 1986-09-22 1986
1 M POL Department of Police ISB Major Crimes Division Fugitive Section Fulltime-Regular Master Police Officer 1988-09-12 1988


.. GENERATED FROM PYTHON SOURCE LINES 247-249 We can observe that there is a ``date_first_hired`` which is time-based. Now, let us apply :func:`skore.train_test_split` on this data: .. GENERATED FROM PYTHON SOURCE LINES 251-257 .. code-block:: Python import skore _ = skore.train_test_split( X=X_employee, y=y_employee, random_state=0, shuffle=False, as_dict=True ) .. rst-class:: sphx-glr-script-out .. code-block:: none ╭─────────────────────────────── TimeBasedColumnWarning ───────────────────────────────╮ │ We detected some time-based columns (column "date_first_hired") in your data. We │ │ recommend using scikit-learn's TimeSeriesSplit instead of train_test_split. │ │ Otherwise you might train on future data to predict the past, or get inflated model │ │ performance evaluation because natural drift will not be taken into account. │ ╰──────────────────────────────────────────────────────────────────────────────────────╯ .. GENERATED FROM PYTHON SOURCE LINES 258-261 We get a ``TimeBasedColumnWarning`` advising us to use :class:`sklearn.model_selection.TimeSeriesSplit` instead! Indeed, we should not shuffle time-ordered data! .. GENERATED FROM PYTHON SOURCE LINES 263-268 .. seealso:: More methodological advice is available. For more information about the motivation and usage of :func:`skore.train_test_split`, see :ref:`example_train_test_split`. .. GENERATED FROM PYTHON SOURCE LINES 270-275 Tracking: skore project ======================= Another key feature of skore is its :class:`~skore.Project` that allows us to store and retrieve :class:`~skore.EstimatorReport` objects. .. GENERATED FROM PYTHON SOURCE LINES 277-279 Setup: creating and loading a skore project ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. GENERATED FROM PYTHON SOURCE LINES 281-282 Let us start by creating a skore project named ``my_project``: .. GENERATED FROM PYTHON SOURCE LINES 284-287 .. code-block:: Python my_project = skore.Project("my_project") .. GENERATED FROM PYTHON SOURCE LINES 295-300 Storing some reports in our project ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Now that the project exists, we can store some useful reports in it using :func:`~skore.Project.put`, with a key-value convention. .. GENERATED FROM PYTHON SOURCE LINES 302-304 Let us store the estimator reports of the random forest and the gradient boosting to help us track our experiments: .. GENERATED FROM PYTHON SOURCE LINES 306-309 .. code-block:: Python my_project.put("estimator_report", rf_report) my_project.put("estimator_report", gb_report) .. GENERATED FROM PYTHON SOURCE LINES 310-312 Retrieving our stored reports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. GENERATED FROM PYTHON SOURCE LINES 314-316 Now, let us retrieve the data that we just stored using a :class:`~skore.project.summary.Summary` object: .. GENERATED FROM PYTHON SOURCE LINES 318-321 .. code-block:: Python summary = my_project.summarize() print(type(summary)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 322-323 We can retrieve the complete list of stored reports: .. GENERATED FROM PYTHON SOURCE LINES 325-330 .. code-block:: Python from pprint import pprint reports_get = summary.reports() pprint(reports_get) .. rst-class:: sphx-glr-script-out .. code-block:: none [EstimatorReport(estimator=RandomForestClassifier(random_state=0), ...), EstimatorReport(estimator=GradientBoostingClassifier(random_state=0), ...)] .. GENERATED FROM PYTHON SOURCE LINES 331-332 For example, we can compare the stored reports: .. GENERATED FROM PYTHON SOURCE LINES 334-337 .. code-block:: Python comparator = ComparisonReport(reports=reports_get) comparator.metrics.summarize(pos_label=1, indicator_favorability=True).frame() .. raw:: html
Estimator RandomForestClassifier GradientBoostingClassifier Favorability
Metric Label / Average
Precision 0 0.878468 0.901370 (↗︎)
1 0.696411 0.707150 (↗︎)
2 0.730337 0.723632 (↗︎)
Recall 0 0.768786 0.760694 (↗︎)
1 0.850711 0.855450 (↗︎)
2 0.657396 0.685209 (↗︎)
ROC AUC 0 0.945966 0.946672 (↗︎)
1 0.906700 0.907682 (↗︎)
2 0.868317 0.869390 (↗︎)
Log loss 0.550382 0.553147 (↘︎)
Fit time (s) 3.443031 18.228429 (↘︎)
Predict time (s) 0.032669 0.010364 (↘︎)


.. GENERATED FROM PYTHON SOURCE LINES 338-340 We can retrieve any accessor of our stored estimator reports, for example the timings from the first estimator report: .. GENERATED FROM PYTHON SOURCE LINES 342-344 .. code-block:: Python reports_get[0].metrics.timings() .. rst-class:: sphx-glr-script-out .. code-block:: none {'fit_time': 3.443031195000003, 'predict_time_test': 0.032668562999987216} .. GENERATED FROM PYTHON SOURCE LINES 345-349 But what if instead of having stored only 2 estimators reports, we had a dozen or even a few hundreds over several months of experimenting? We would need to navigate through our stored estimator reports. For that, the skore project provides a convenient search feature. .. GENERATED FROM PYTHON SOURCE LINES 351-371 Searching through our stored reports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Using the interactive widget """""""""""""""""""""""""""" If rendered in a Jupyter notebook, ``summary`` would render an interactive parallel coordinate plot to search for your preferred model based on some metrics. Here is a screenshot: .. image:: /_static/images/screenshot_getting_started.png :alt: Screenshot of the widget in a Jupyter notebook How to use the widget? You select the estimator(s) you are interested in by clicking on the plot and the metric(s) you are interested in by checking them. Then, using the python API, we can retrieve the *corresponding* list of stored reports: .. code:: python summary.reports() .. GENERATED FROM PYTHON SOURCE LINES 373-377 Using the Python API """""""""""""""""""" Alternatively, this search feature can be performed using the Python API. .. GENERATED FROM PYTHON SOURCE LINES 379-380 We can perform some queries on our stored data using the following keys: .. GENERATED FROM PYTHON SOURCE LINES 382-384 .. code-block:: Python pprint(summary.keys()) .. rst-class:: sphx-glr-script-out .. code-block:: none Index(['run_id', 'key', 'date', 'learner', 'ml_task', 'report_type', 'dataset', 'rmse', 'log_loss', 'roc_auc', 'fit_time', 'predict_time', 'rmse_mean', 'log_loss_mean', 'roc_auc_mean', 'fit_time_mean', 'predict_time_mean'], dtype='object') .. GENERATED FROM PYTHON SOURCE LINES 385-387 For example, we can query all the estimators corresponding to a :class:`~sklearn.ensemble.RandomForestClassifier`: .. GENERATED FROM PYTHON SOURCE LINES 389-394 .. code-block:: Python report_search_rf = summary.query( "learner.str.contains('RandomForestClassifier')" ).reports() pprint(report_search_rf) .. rst-class:: sphx-glr-script-out .. code-block:: none [EstimatorReport(estimator=RandomForestClassifier(random_state=0), ...)] .. GENERATED FROM PYTHON SOURCE LINES 395-397 Or, we can query all the estimator reports corresponding to a classification task: .. GENERATED FROM PYTHON SOURCE LINES 399-402 .. code-block:: Python report_search_clf = summary.query("ml_task.str.contains('classification')").reports() pprint(report_search_clf) .. rst-class:: sphx-glr-script-out .. code-block:: none [EstimatorReport(estimator=RandomForestClassifier(random_state=0), ...), EstimatorReport(estimator=GradientBoostingClassifier(random_state=0), ...)] .. GENERATED FROM PYTHON SOURCE LINES 406-414 .. admonition:: Stay tuned! These are only the initial features: skore is a work in progress and aims to be an end-to-end library for data scientists. Feedbacks are welcome: please feel free to join our `Discord `_ or `create an issue `_. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 46.293 seconds) .. _sphx_glr_download_auto_examples_getting_started_plot_skore_getting_started.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_skore_getting_started.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_skore_getting_started.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_skore_getting_started.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_