Skore: getting started

This guide illustrates how to use skore through a complete machine learning workflow for binary classification:

1. Set up a proper experiment with training and test data

2. Develop and evaluate multiple models using cross-validation

3. Compare models to select the best one

4. Validate the final model on held-out data

5. Track and organize your machine learning results

Throughout this guide, we will see how skore helps you:

• Avoid common pitfalls with smart diagnostics

• Quickly get rich insights into model performance

• Organize and track your experiments

Storing reports in Skore Hub

At the end of this example, we send the reports in Skore Hub (https://skore.probabl.ai/) that is a platform for storing, sharing and exploring your machine learning reports.

To run this example and push in your own Skore Hub workspace and project, you can run this example with the following command:

WORKSPACE=<workspace> PROJECT=<project> python plot_getting_started.py

In this gallery, we are going to push the different reports into a public workspace.

Setting up our classification problem

Let’s start by loading the “toxicity” dataset, a classification problem where we classify tweets as “toxic” or “not toxic”.

from skrub import TableReport, datasets

toxicity = datasets.fetch_toxicity()
X, y = toxicity.X, toxicity.y
TableReport(toxicity.toxicity)

Downloading 'toxicity_v1' from https://github.com/skrub-data/skrub-data-files/raw/refs/heads/main/toxicity_v1.zip (attempt 1/3)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We create a held-out test set to evaluate our final model once we are done experimenting.

from sklearn.model_selection import train_test_split

X_experiment, X_holdout, y_experiment, y_holdout = train_test_split(
    X, y, random_state=0
)

Model development with cross-validation

We will investigate two different families of models using cross-validation.

1. A LogisticRegression which is a linear model

2. A RandomForestClassifier which is a more powerful model.

In both cases, we rely on skrub.tabular_pipeline() to choose the proper preprocessing depending on the kind of model.

Cross-validation is necessary to get a more reliable estimate of model performance. skore makes it easy through skore.CrossValidationReport.

Model no. 1: logistic regression with preprocessing

Our first model will be a linear model, with automatic preprocessing of the text feature. Under the hood, skrub’s TableVectorizer will adapt the preprocessing based on our choice to use a linear model.

from sklearn.linear_model import LogisticRegression
from skrub import tabular_pipeline

logistic_regression = tabular_pipeline(LogisticRegression())
logistic_regression

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                ('logisticregression', LogisticRegression())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

We now evaluate our model with cross-validation, using evaluate() with splitter=5 to perform 5-fold cross-validation. This returns a CrossValidationReport object, which can be used to access the performance metrics and other information about the model.

from skore import evaluate

logreg_cv_report = evaluate(
    logistic_regression, X_experiment, y_experiment, pos_label="Toxic", splitter=5
)
logreg_cv_report

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                ('logisticregression', LogisticRegression())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

0 issue(s), 1 tip(s), 1 passed, 0 ignored.

A report will quickly show important information regarding the performance of the model, the dataset used and the architecture of the model. This information is only a quick overview and one can dig deeper into the report to get more information.

Indeed, Skore reports allow to structure the statistical information we look for when experimenting with predictive models. First, the help() method shows us all its available methods and attributes, with the knowledge that our model was trained for classification:

logreg_cv_report.help()

For example, we can examine the training data, which excludes the held-out data:

logreg_cv_report.data.summarize()

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Additionally we can run automatic checks on the model and get a summary of the findings:

logreg_cv_report.checks.summarize()

But we can also quickly get an overview of the performance of our model, using summarize():

logreg_metrics = logreg_cv_report.metrics.summarize()
logreg_metrics.frame(favorability=True)

	LogisticRegression		Favorability
	mean	std
Metric
Score	0.822667	0.032863	(↗︎)
Accuracy	0.822667	0.032863	(↗︎)
Precision	0.821093	0.026791	(↗︎)
Recall	0.827895	0.047379	(↗︎)
ROC AUC	0.903742	0.028689	(↗︎)
Log loss	0.387494	0.040477	(↘︎)
Brier score	0.124589	0.016317	(↘︎)
Fit time (s)	0.174646	0.007059	(↘︎)
Predict time (s)	0.028925	0.000578	(↘︎)

Note

favorability=True adds a column showing whether higher or lower metric values are better.

In addition to the summary of metrics, skore provides more advanced statistical information such as the precision-recall curve:

precision_recall = logreg_cv_report.metrics.precision_recall()
precision_recall.help()

Note

The output of precision_recall() is a Display object. This is a common pattern in skore which allows us to access the information in several ways.

We can visualize the critical information as a plot, with only a few lines of code:

_ = precision_recall.plot()

Or we can access the raw information as a dataframe if additional analysis is needed:

precision_recall.frame()

	split	threshold	precision	recall
0	0	0.009867	0.506667	1.000000
1	0	0.018259	0.520548	1.000000
2	0	0.028762	0.517241	0.986842
3	0	0.073037	0.600000	0.986842
4	0	0.074317	0.596774	0.973684
...	...	...	...	...
471	4	0.994304	1.000000	0.066667
472	4	0.995458	1.000000	0.053333
473	4	0.997181	1.000000	0.040000
474	4	0.999078	1.000000	0.026667
475	4	0.999713	1.000000	0.013333

476 rows × 4 columns

As another example, we can plot the confusion matrix with the same consistent API:

confusion_matrix = logreg_cv_report.metrics.confusion_matrix()
_ = confusion_matrix.plot()

Skore also provides utilities to inspect models. Since our model is a linear model, we can study the importance that it gives to each feature:

coefficients = logreg_cv_report.inspection.coefficients()
coefficients.frame()

	feature	coefficient_mean	coefficient_std
0	Intercept	-0.228173	0.057426
1	text_00	0.491118	0.061281
2	text_01	0.619398	0.203769
3	text_02	-0.012838	2.277056
4	text_03	0.284167	1.028327
5	text_04	1.395130	0.951667
6	text_05	0.142982	0.837875
7	text_06	0.369620	0.349657
8	text_07	0.240669	0.226409
9	text_08	0.007752	0.768454
10	text_09	0.401025	0.501284
11	text_10	-0.375868	0.319794
12	text_11	0.037939	0.126357
13	text_12	-0.216975	0.486864
14	text_13	0.088168	0.281896
15	text_14	-0.277356	0.262769
16	text_15	0.029695	0.287129
17	text_16	-0.079262	0.193640
18	text_17	0.096753	0.371877
19	text_18	0.047776	0.550933
20	text_19	-0.057647	0.435149
21	text_20	-0.165289	0.244894
22	text_21	-0.193380	0.143431
23	text_22	0.127968	0.154432
24	text_23	0.207588	0.350015
25	text_24	0.055779	0.200928
26	text_25	-0.019072	0.338113
27	text_26	-0.168036	0.112960
28	text_27	0.335682	0.318793
29	text_28	-0.064570	0.422299
30	text_29	-0.038096	0.120775

_ = coefficients.plot(select_k=15)

Model no. 2: Random forest

Now, we cross-validate a more powerful model using RandomForestClassifier. Again, we rely on tabular_pipeline() to perform the appropriate preprocessing to use with this model.

from sklearn.ensemble import RandomForestClassifier

random_forest = tabular_pipeline(RandomForestClassifier(random_state=0))
random_forest

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                unknown_value=-1))),
                ('randomforestclassifier',
                 RandomForestClassifier(random_state=0))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

rf_cv_report = evaluate(
    random_forest, X_experiment, y_experiment, pos_label="Toxic", splitter=5
)
rf_cv_report

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                unknown_value=-1))),
                ('randomforestclassifier',
                 RandomForestClassifier(random_state=0))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

1 issue(s), 1 tip(s), 1 passed, 0 ignored.

rf_cv_report.checks.summarize()

We will now compare this new model with the previous one.

Comparing our models

Now that we have our two models, we need to decide which one should go into production. We can compare them with the compare() function that returns a ComparisonReport:

from skore import compare

comparison = compare(
    {
        "logistic regression": logreg_cv_report,
        "random forest": rf_cv_report,
    },
)
comparison

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                ('logisticregression', LogisticRegression())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

1 issue(s), 1 tip(s), 1 passed, 0 ignored.

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                unknown_value=-1))),
                ('randomforestclassifier',
                 RandomForestClassifier(random_state=0))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

3 issue(s), 1 tip(s), 1 passed, 0 ignored.

This report follows the same API as CrossValidationReport:

comparison.help()

We have access to the same tools to perform statistical analysis and compare both models:

comparison_metrics = comparison.metrics.summarize()
comparison_metrics.frame(favorability=True)

	mean		std		Favorability
Estimator	logistic regression	random forest	logistic regression	random forest
Metric
Score	0.822667	0.793333	0.032863	0.038586	(↗︎)
Accuracy	0.822667	0.793333	0.032863	0.038586	(↗︎)
Precision	0.821093	0.819822	0.026791	0.056171	(↗︎)
Recall	0.827895	0.759404	0.047379	0.042142	(↗︎)
ROC AUC	0.903742	0.877517	0.028689	0.042379	(↗︎)
Log loss	0.387494	0.459921	0.040477	0.046979	(↘︎)
Brier score	0.124589	0.146664	0.016317	0.019588	(↘︎)
Fit time (s)	0.174646	0.389348	0.007059	0.006218	(↘︎)
Predict time (s)	0.028925	0.038924	0.000578	0.000364	(↘︎)

_ = comparison.metrics.precision_recall().plot()

Based on the previous tables and plots, it seems that the RandomForestClassifier model has slightly worse performance due to overfitting on this small dataset. We make the choice to deploy the linear model to make a comparison with the coefficients study shown earlier.

Final model evaluation on held-out data

Now that we have chosen to deploy the linear model, we will train it on the full experiment set and evaluate it on our held-out data: training on more data should help performance and we can also validate that our model generalizes well to new data. This can be done in one step with create_estimator_report().

final_report = comparison.create_estimator_report(
    report_key="logistic regression", X_test=X_holdout, y_test=y_holdout
)
final_report

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                ('logisticregression', LogisticRegression())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

0 issue(s), 1 tip(s), 4 passed, 0 ignored.

This returns a EstimatorReport which has a similar API to the other report classes:

final_metrics = final_report.metrics.summarize()
final_metrics.frame()

	LogisticRegression
Metric
Score	0.836000
Accuracy	0.836000
Precision	0.836066
Recall	0.829268
ROC AUC	0.918251
Log loss	0.359038
Brier score	0.115043
Fit time (s)	0.201293
Predict time (s)	0.037999

_ = final_report.metrics.confusion_matrix().plot()

We can easily combine the results of the previous cross-validation together with the evaluation on the held-out dataset, since the two are accessible as dataframes. This way, we can check if our chosen model meets the expectations we set during the experiment phase.

import pandas as pd

pd.concat(
    [final_metrics.frame(), logreg_cv_report.metrics.summarize().frame()],
    axis="columns",
)

	LogisticRegression	(LogisticRegression, mean)	(LogisticRegression, std)
Metric
Score	0.836000	0.822667	0.032863
Accuracy	0.836000	0.822667	0.032863
Precision	0.836066	0.821093	0.026791
Recall	0.829268	0.827895	0.047379
ROC AUC	0.918251	0.903742	0.028689
Log loss	0.359038	0.387494	0.040477
Brier score	0.115043	0.124589	0.016317
Fit time (s)	0.201293	0.174646	0.007059
Predict time (s)	0.037999	0.028925	0.000578

As expected, our final model gets better performance, likely thanks to the larger training set.

Our final sanity check is to compare the features considered most impactful between our final model and the cross-validation:

final_coefficients = final_report.inspection.coefficients()
cv_coefficients = logreg_cv_report.inspection.coefficients()

features_final_coefficients = final_coefficients.frame(select_k=15)["feature"]
features_cv_coefficients = cv_coefficients.frame(select_k=15)["feature"]

print(
    f"Most important features available in both models: "
    f"{set(features_final_coefficients).intersection(set(features_cv_coefficients))}"
)

print(
    f"Most important features available in final model but not in cross-validation: "
    f"{set(features_final_coefficients).difference(set(features_cv_coefficients))}"
)

Most important features available in both models: {'text_06', 'text_02', 'text_09', 'text_12', 'text_23', 'text_00', 'text_08', 'text_04', 'text_01'}
Most important features available in final model but not in cross-validation: {'text_28', 'text_20', 'text_16', 'text_24', 'text_26', 'text_25'}

We can further check if there is a drastic difference in the ordering by plotting those features with the largest absolute coefficients.

final_coefficients.plot(select_k=15, sorting_order="descending")
_ = cv_coefficients.plot(select_k=15, sorting_order="descending")

• 
• 

They seem very similar, so we are done!

Tracking our work with a skore Project

Now that we have completed our modeling workflow, we should store our models in a safe place for future work. Indeed, if this research notebook were modified, we would no longer be able to relate the current production model to the code that generated it.

We can use a skore.Project to keep track of our experiments. This makes it easy to organize, retrieve, and compare models over time.

Usually this would be done as you go along the model development, but in the interest of simplicity we kept this until the end.

We are using Skore Hub (https://skore.probabl.ai/) to store and review our reports.

Note

Here, we are using Skore Hub to store and analyze the reports that we computed. Note that you can store reports as well locally using mode="local" when creating or loading projects via skore.Project.

from skore import login

login()

╭───────────────────────────────── Login to Skore Hub ─────────────────────────────────╮
│                                                                                      │
│                        Successfully logged in, using API key.                        │
│                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────╯

We load or create a hub project:

project = Project(f"{WORKSPACE}/{PROJECT}", mode="hub")

We store our reports with descriptive keys:

project.put("logreg_cv", logreg_cv_report)

  Putting logreg_cv 0:01:41
Consult your report at
https://skore.probabl.ai/skore/example-getting-started-dev/cross-validations/27257

project.put("rf_cv", rf_cv_report)

  Putting rf_cv 0:01:38
Consult your report at
https://skore.probabl.ai/skore/example-getting-started-dev/cross-validations/27263

In this example, we created a read-only Skore Hub project that you can visit by clicking on the link above and explore the reports.

Now we can retrieve a summary of our stored reports:

summary = project.summarize()
summary

Note

summarize() returns a Summary object. In a Jupyter environment it renders as an interactive table where you can filter rows and pick reports across the different views; the selection produces a query string ready to pass to query() so you can recover exactly those reports.

Once you filtered the summary (e.g. to keep only the cross-validation reports), if you now call compare(), you get only the CrossValidationReport objects, which you can directly put in the form of a ComparisonReport:

new_report = summary.query('report_type == "cross-validation"').compare(
    return_as="report"
)
new_report

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('squashingscaler', SquashingScaler(max_absolute_value=5)),
                ('logisticregression', LogisticRegression())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

1 issue(s), 1 tip(s), 1 passed, 0 ignored.

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                unknown_value=-1))),
                ('randomforestclassifier',
                 RandomForestClassifier(random_state=0))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

3 issue(s), 1 tip(s), 1 passed, 0 ignored.

Stay tuned!

This is only the beginning for skore. We welcome your feedback and ideas to make it the best tool for end-to-end data science.

Key benefits of using skore in your ML workflow:

• Standardized evaluation and comparison of models

• Rich visualizations and diagnostics

• Organized experiment tracking

• Seamless integration with scikit-learn

Feel free to join our community on Discord or create an issue.