Cache mechanism

This example shows how EstimatorReport and CrossValidationReport use caching to speed up computations.

Generating some data

In this toy example, we create a large synthetic classification dataset that will let us see speed improvements easily.

import pandas as pd
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=150_000, return_X_y=True)
X = pd.DataFrame(X, columns=[str(i) for i in range(X.shape[1])])

Here is what the training data looks like:

from skrub import TableReport

TableReport(X)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

And the target training data:

TableReport(y)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We build a model using skrub.tabular_pipeline(): it is a simple predictive model that also performs basic feature engineering.

from skrub import tabular_pipeline

model = tabular_pipeline("classifier")
model

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=ToCategorical())),
                ('histgradientboostingclassifier',
                 HistGradientBoostingClassifier())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Caching the predictions for fast metric computation

Let’s explore how EstimatorReport uses caching to speed up predictions.

from skore import evaluate

report = evaluate(model, X, y, pos_label=1)
report.help()

We compute the accuracy on our test set and measure how long it takes:

import time

start = time.time()
report.metrics.accuracy()
end = time.time()
print(f"Time taken: {end - start:.3f} seconds")

Time taken: 0.003 seconds

For comparison, here’s how scikit-learn computes the same accuracy score:

from sklearn.metrics import accuracy_score

start = time.time()
accuracy_score(report.y_test, report.estimator_.predict(report.X_test))
end = time.time()
print(f"Time taken: {end - start:.2f} seconds")

Time taken: 0.20 seconds

skore outputs the result much faster than scikit-learn. How can this be? The answer lies in the EstimatorReport’s state. When the EstimatorReport is created, it computes the model predictions, and caches them:

report._cache

{('report', 'test', 'decision_function', None): array([[-4.60866105,  4.60866105],
       [ 5.00218591, -5.00218591],
       [-3.22904507,  3.22904507],
       ...,
       [-4.01535656,  4.01535656],
       [-4.36214808,  4.36214808],
       [ 5.20118691, -5.20118691]], shape=(30000, 2)), ('report', 'test', 'predict', None): array([1, 0, 1, ..., 1, 1, 0], shape=(30000,)), ('report', 'test', 'predict_proba', None): array([[0.00986683, 0.99013317],
       [0.99332167, 0.00667833],
       [0.03808722, 0.96191278],
       ...,
       [0.01771697, 0.98228303],
       [0.01259043, 0.98740957],
       [0.99452017, 0.00547983]], shape=(30000, 2)), ('report', 'test', 'predict_log_proba', None): array([[-4.61857688, -0.00991583],
       [-0.00670073, -5.00888665],
       [-3.26787657, -0.03883149],
       ...,
       [-4.03323236, -0.01787579],
       [-4.37481844, -0.01267036],
       [-0.0054949 , -5.20668181]], shape=(30000, 2)), ('metrics', 'test', 'accuracy', ('mapping', ())): 0.9458666666666666}

The cache stores predictions by type and data source. This means that computing metrics that use the same type of predictions will be faster.

Caching with CrossValidationReport

Here we will demonstrate that CrossValidationReport also benefits from caching.

report = evaluate(model, X=X, y=y, splitter=3, n_jobs=3)
report.help()

A CrossValidationReport is essentially a list of EstimatorReport, one for each split, so caching on the splits makes the calculation on the CrossValidationReport faster as well.

start = time.time()
report.metrics.summarize().frame()
end = time.time()
print(f"Time taken: {end - start:.2f} seconds")

Time taken: 0.13 seconds

The subsequent calls are even faster because the metrics themselves are cached:

start = time.time()
report.metrics.summarize().frame()
end = time.time()
print(f"Time taken: {end - start:.2f} seconds")

Time taken: 0.02 seconds

By keeping the estimator together with the data, we are able to trade off some memory space for faster operations.