{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# SKD013 - Train-test overlap in time series\n\nThis example demonstrates mitigations when check\n`SKD013 <skd013-train-test-time-overlap>` fires on temporal data. The\ncheck compares datetime columns in train and test folds and flags overlap when\nthe latest training timestamp is not strictly before the earliest test\ntimestamp.\n\nMitigations from the `automated_checks` user guide:\n\n- use a time-based splitter such as\n  :class:`~sklearn.model_selection.TimeSeriesSplit` or similar.\n\nWe use the employee salaries dataset ordered by hire date and predict whether\nsalary exceeds the median. The goal is to evaluate on future hires only so\ntest scores reflect forward-looking performance.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Load the employee salaries dataset\n\nRows are sorted by ``date_first_hired``. We expose hire dates as a pandas\n``timestamp`` column because SKD013 requires a datetime dtype. Shuffling\nbefore a hold-out split mixes later hires into training; that is the pattern\nSKD013 is designed to catch. Use full\n:meth:`~skore.EstimatorReport.checks.summarize` on the trigger;\n``fast_mode=True`` on fix cells.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import pandas as pd\nfrom skrub.datasets import fetch_employee_salaries\n\ndataset = fetch_employee_salaries()\ndf = dataset.X.copy()\ndf[\"current_annual_salary\"] = dataset.y\ndf[\"timestamp\"] = pd.to_datetime(df[\"date_first_hired\"])\ndf = df.sort_values(\"timestamp\").reset_index(drop=True)\n\ny = (df[\"current_annual_salary\"] > df[\"current_annual_salary\"].median()).astype(int)\nX = df.drop(columns=[\"current_annual_salary\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        ":class:`~skrub.TableReport` confirms chronological ordering and mixed HR\nfeatures.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport\n\nTableReport(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The binary target marks above-median earners, a classification view of the\nsalary column.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "TableReport(y.to_frame(name=\"high_earner\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Trigger SKD013 - shuffled train/test split\n\n:class:`~skore.TrainTestSplit` with ``shuffle=True`` randomizes row order\nbefore cutting folds, so future timestamps land in training. Fit a tabular\nclassifier and summarize checks.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import TrainTestSplit, evaluate\nfrom skrub import tabular_pipeline\n\nsplitter_shuffled = TrainTestSplit(random_state=42, shuffle=True)\n\nreport = evaluate(\n    tabular_pipeline(\"classifier\"),\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter_shuffled,\n)\nreport"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD013 should list ``timestamp`` as overlapping between train and test.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.checks.summarize()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report.metrics.summarize(data_source=\"both\").frame(favorability=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Optimistic scores under shuffled splits are a leakage artifact; SKD013 forces\nyou to respect time ordering before trusting metrics.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Chronological hold-out\n\n``shuffle=False`` keeps the test block as the latest rows in the table. No\ntraining row should carry a timestamp on or after the earliest test hire.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "splitter_chrono = TrainTestSplit(random_state=42, shuffle=False)\n\nreport_chrono = evaluate(\n    tabular_pipeline(\"classifier\"),\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter_chrono,\n)\nreport_chrono"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD013 should be absent; test rows are strictly after train rows.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_chrono.checks.summarize(fast_mode=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_chrono.metrics.summarize(data_source=\"both\").frame(favorability=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "A simple chronological hold-out is often enough for deployment monitoring\nwhen you score on the most recent period.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# TimeSeriesSplit for cross-validated evaluation\n\n:class:`~sklearn.model_selection.TimeSeriesSplit` trains on past rows and\ntests on the next chunk in each fold. Many employees share the same hire\ndate, so a default split can still place the same calendar day in train and\ntest at the fold boundary (SKD013 uses ``>=``). A small ``gap`` skips rows\nbetween folds and clears that tie. Early folds are small and unrelated checks\nsuch as SKD008 may warn on encoded features; we ignore SKD008 here to focus\non temporal validity.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import TimeSeriesSplit\n\nsplitter_tscv = TimeSeriesSplit(n_splits=5, gap=50)\n\nreport_tscv = evaluate(\n    tabular_pipeline(\"classifier\"),\n    X=X,\n    y=y,\n    pos_label=1,\n    splitter=splitter_tscv,\n)\nreport_tscv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "SKD013 should be absent; no fold trains on timestamps on or after its test\nblock.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tscv.checks.summarize(fast_mode=True, ignore=[\"SKD008\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "report_tscv.metrics.summarize(data_source=\"both\").frame(aggregate=\"mean\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Time-series cross-validation estimates stability across multiple forward\nwindows, the right tool when a single hold-out is too noisy.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Conclusion\n\nSKD013 protects against training on the future. In this walkthrough,\ndisabling shuffle and adopting ``TimeSeriesSplit`` aligned evaluation with\nhow salary models are deployed over newly hired employees. Always pair\ntemporal splits with features available at prediction time.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}