SKD013 - Train-test overlap in time series

This example demonstrates mitigations when check SKD013 fires on temporal data. The check compares datetime columns in train and test folds and flags overlap when the latest training timestamp is not strictly before the earliest test timestamp.

Mitigations from the Automated checks user guide:

• use a time-based splitter such as TimeSeriesSplit or similar.

We use the employee salaries dataset ordered by hire date and predict whether salary exceeds the median. The goal is to evaluate on future hires only so test scores reflect forward-looking performance.

Load the employee salaries dataset

Rows are sorted by date_first_hired. We expose hire dates as a pandas timestamp column because SKD013 requires a datetime dtype. Shuffling before a hold-out split mixes later hires into training; that is the pattern SKD013 is designed to catch. Use full summarize() on the trigger; fast_mode=True on fix cells.

import pandas as pd
from skrub.datasets import fetch_employee_salaries

dataset = fetch_employee_salaries()
df = dataset.X.copy()
df["current_annual_salary"] = dataset.y
df["timestamp"] = pd.to_datetime(df["date_first_hired"])
df = df.sort_values("timestamp").reset_index(drop=True)

y = (df["current_annual_salary"] > df["current_annual_salary"].median()).astype(int)
X = df.drop(columns=["current_annual_salary"])

TableReport confirms chronological ordering and mixed HR features.

from skrub import TableReport

TableReport(X)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

The binary target marks above-median earners, a classification view of the salary column.

TableReport(y.to_frame(name="high_earner"))

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Trigger SKD013 - shuffled train/test split

TrainTestSplit with shuffle=True randomizes row order before cutting folds, so future timestamps land in training. Fit a tabular classifier and summarize checks.

from skore import TrainTestSplit, evaluate
from skrub import tabular_pipeline

splitter_shuffled = TrainTestSplit(random_state=42, shuffle=True)

report = evaluate(
    tabular_pipeline("classifier"),
    X=X,
    y=y,
    pos_label=1,
    splitter=splitter_shuffled,
)
report

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=ToCategorical())),
                ('histgradientboostingclassifier',
                 HistGradientBoostingClassifier())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

SKD013 should list timestamp as overlapping between train and test.

report.checks.summarize()

report.metrics.summarize(data_source="both").frame(favorability=True)

	HistGradientBoostingClassifier (train)	HistGradientBoostingClassifier (test)	favorability
metric
accuracy	0.986589	0.942579	(↗︎)
precision	0.989133	0.945856	(↗︎)
recall	0.984054	0.937568	(↗︎)
roc_auc	0.999216	0.988270	(↗︎)
log_loss	0.053973	0.141108	(↘︎)
brier_score	0.012377	0.043155	(↘︎)
fit_time	2.127929	2.127929	(↘︎)
predict_time	0.812125	0.252934	(↘︎)

Optimistic scores under shuffled splits are a leakage artifact; SKD013 forces you to respect time ordering before trusting metrics.

Chronological hold-out

shuffle=False keeps the test block as the latest rows in the table. No training row should carry a timestamp on or after the earliest test hire.

splitter_chrono = TrainTestSplit(random_state=42, shuffle=False)

report_chrono = evaluate(
    tabular_pipeline("classifier"),
    X=X,
    y=y,
    pos_label=1,
    splitter=splitter_chrono,
)
report_chrono

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=ToCategorical())),
                ('histgradientboostingclassifier',
                 HistGradientBoostingClassifier())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

SKD013 should be absent; test rows are strictly after train rows.

report_chrono.checks.summarize(fast_mode=True)

report_chrono.metrics.summarize(data_source="both").frame(favorability=True)

	HistGradientBoostingClassifier (train)	HistGradientBoostingClassifier (test)	favorability
metric
accuracy	0.987673	0.929577	(↗︎)
precision	0.991280	0.805479	(↗︎)
recall	0.987324	0.832861	(↗︎)
roc_auc	0.999099	0.974609	(↗︎)
log_loss	0.054897	0.165643	(↘︎)
brier_score	0.012701	0.051366	(↘︎)
fit_time	2.283330	2.283330	(↘︎)
predict_time	0.812014	0.290345	(↘︎)

A simple chronological hold-out is often enough for deployment monitoring when you score on the most recent period.

TimeSeriesSplit for cross-validated evaluation

TimeSeriesSplit trains on past rows and tests on the next chunk in each fold. Many employees share the same hire date, so a default split can still place the same calendar day in train and test at the fold boundary (SKD013 uses >=). A small gap skips rows between folds and clears that tie. Early folds are small and unrelated checks such as SKD008 may warn on encoded features; we ignore SKD008 here to focus on temporal validity.

from sklearn.model_selection import TimeSeriesSplit

splitter_tscv = TimeSeriesSplit(n_splits=5, gap=50)

report_tscv = evaluate(
    tabular_pipeline("classifier"),
    X=X,
    y=y,
    pos_label=1,
    splitter=splitter_tscv,
)
report_tscv

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=ToCategorical())),
                ('histgradientboostingclassifier',
                 HistGradientBoostingClassifier())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

SKD013 should be absent; no fold trains on timestamps on or after its test block.

report_tscv.checks.summarize(fast_mode=True, ignore=["SKD008"])

report_tscv.metrics.summarize(data_source="both").frame(aggregate="mean")

	histgradientboostingclassifier_(train)_mean	histgradientboostingclassifier_(test)_mean
metric
accuracy	0.995442	0.904551
precision	0.996773	0.827716
recall	0.995608	0.917845
roc_auc	0.999732	0.946625
log_loss	0.023278	0.331034
brier_score	0.004821	0.079583
fit_time	1.667619	1.667619
predict_time	0.573863	0.275924

Time-series cross-validation estimates stability across multiple forward windows, the right tool when a single hold-out is too noisy.

Conclusion

SKD013 protects against training on the future. In this walkthrough, disabling shuffle and adopting TimeSeriesSplit aligned evaluation with how salary models are deployed over newly hired employees. Always pair temporal splits with features available at prediction time.