{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n\n# SKD008 - Highly correlated input features\n\nThis example walks through mitigations when check\n`SKD008 ` fires because numeric columns are\nnearly redundant. The check computes pairwise Spearman correlation on training\ninputs and flags pairs with $|\u03c1| > 0.9$.\n\nWe showcase the following mitigations from the `automated_checks` user\nguide:\n\n- remove or combine redundant features,\n- use L1/L2 regularization models as ``Ridge`` or ``Lasso`` in regression or a\n penalized ``LogisticRegression``,\n- group correlated features before inspecting importance.\n\nWe use the breast cancer Wisconsin dataset, where radius, perimeter, and area\nmeasurements are almost linearly related. The goal is to simplify the feature\ntable without losing signal.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load the breast cancer Wisconsin dataset\n\nThe dataset describes cell nuclei with 30 numeric features pertaining to\ncell size, shape, and texture. Many of these features are correlated;\nthe check should help detect this. Class ``0`` is malignant (our positive\nlabel of interest).\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n\nX, y = load_breast_cancer(as_frame=True, return_X_y=True)\npos_label = 0 # malignant" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The target is moderately imbalanced but easy to separate; anyway, our concern\nin this example is collinearity of features.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from skrub import TableReport\n\nTableReport(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us use a stratified :class:`~skore.TrainTestSplit` so both classes appear\nin train and test.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from skore import TrainTestSplit\n\nsplitter = TrainTestSplit(random_state=42, stratify=y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Trigger SKD008: full feature set\n\nA gradient boosting classifier tolerates correlated inputs, but SKD008 still\ninspects the training matrix. Let us fit on the full table, then summarize\nchecks.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.ensemble import HistGradientBoostingClassifier\nfrom skore import evaluate\n\nreport = evaluate(\n HistGradientBoostingClassifier(random_state=42),\n X=X,\n y=y,\n splitter=splitter,\n pos_label=pos_label,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SKD008 gives the number of highly correlated feature pairs on the training\nfold.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Investigate correlated pairs\n\nThe check prompts us to look more closely at the data. The\n:class:`~skrub.TableReport` \"Associations\" tab indeed shows many highly\ncorrelated feature pairs (for instance radius, perimeter, and area within\neach size block).\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "TableReport(X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Remove redundant features\n\nOne way to satisfy the check is to drop some of the correlated features.\nLet us drop:\n\n- perimeter and area within each size block since they are correlated with\n radius,\n- the \"error\" and \"worst\" features,\n- the features highly correlated with ``mean concavity``.\n\nWe put the drop inside a\n:class:`~sklearn.preprocessing.FunctionTransformer` so the same column\nselection is applied on train and test as part of the estimator.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import FunctionTransformer\n\ncols_to_drop = [\n \"mean perimeter\",\n \"mean area\",\n \"mean concave points\",\n \"mean compactness\",\n] + [c for c in X.columns if \"worst\" in c or \"error\" in c]\n\n\ndef drop_redundant_features(X_df):\n return X_df.drop(columns=[c for c in cols_to_drop if c in X_df.columns])\n\n\nmodel_dropped = make_pipeline(\n FunctionTransformer(drop_redundant_features),\n HistGradientBoostingClassifier(random_state=42),\n)\n\nreport_dropped = evaluate(\n model_dropped,\n X=X,\n y=y,\n splitter=splitter,\n pos_label=pos_label,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SKD008 no longer fires.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_dropped.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Group correlated features by clustering\n\nRather than hand-picking groups and averaging them, we follow the same idea\nas the scikit-learn example on\n[permutation importance with multicollinear features](https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance_multicollinear.html):\nhierarchical clustering on Spearman correlations, then keep one feature per\ncluster. We build the linkage on the training fold of the full-feature\nreport so the grouping does not peek at the test set.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from collections import defaultdict\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom scipy.cluster import hierarchy\nfrom scipy.spatial.distance import squareform\nfrom scipy.stats import spearmanr\n\nX_train = report.X_train\ncorr = spearmanr(X_train).correlation\ncorr = (corr + corr.T) / 2\nnp.fill_diagonal(corr, 1)\ndistance_matrix = 1 - np.abs(corr)\ndist_linkage = hierarchy.ward(squareform(distance_matrix))\n\nfig, ax = plt.subplots(figsize=(10, 4))\nhierarchy.dendrogram(\n dist_linkage,\n labels=X_train.columns.to_list(),\n ax=ax,\n leaf_rotation=90,\n)\nax.set_title(\"Hierarchical clustering of features (Spearman distance)\")\nfig.tight_layout()\n_ = fig" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Cutting the dendrogram at distance ``1`` (as in the scikit-learn example)\nyields compact clusters. Inspecting them, we recover familiar blocks such as\nradius / perimeter / area, or the texture triplet \u2014 similar to the hand-built\ngroups one might have written from the Associations tab.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cluster_ids = hierarchy.fcluster(dist_linkage, t=1, criterion=\"distance\")\ncluster_id_to_features = defaultdict(list)\nfor feature_name, cluster_id in zip(X_train.columns, cluster_ids, strict=True):\n cluster_id_to_features[int(cluster_id)].append(feature_name)\n\ncluster_table = (\n pd.Series(\n {\n cid: \", \".join(names)\n for cid, names in sorted(cluster_id_to_features.items())\n },\n name=\"features\",\n )\n .rename_axis(\"cluster\")\n .reset_index()\n)\ncluster_table" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Keep the first feature of each cluster and wrap that selection in the\npipeline.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "selected_features = [names[0] for names in cluster_id_to_features.values()]\nselected_features\n\n\ndef keep_cluster_representatives(X_df, columns=selected_features):\n return X_df.loc[:, columns]\n\n\nmodel_clustered = make_pipeline(\n FunctionTransformer(keep_cluster_representatives),\n HistGradientBoostingClassifier(random_state=42),\n)\n\nreport_clustered = evaluate(\n model_clustered,\n X=X,\n y=y,\n splitter=splitter,\n pos_label=pos_label,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SKD008 no longer fires.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_clustered.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Compare strategies\n\n:func:`~skore.compare` contrasts test metrics for the full, dropped, and\ncluster-selected feature tables on the same stratified split. This dataset is\nreally simple so the metrics are not that different between the three\nstrategies.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from skore import compare\n\ncomparison = compare(\n {\n \"full_features\": report,\n \"dropped_redundant\": report_dropped,\n \"cluster_representatives\": report_clustered,\n }\n)\ncomparison.metrics.summarize(data_source=\"both\").frame()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Note: L1 logistic regression does not clear SKD008\n\n:class:`~sklearn.linear_model.LogisticRegression` with an L1 penalty can\nshrink coefficients of redundant inputs toward zero (a classification\nanalogue of Lasso). SKD008 only inspects the input matrix, so the check still\nfires; once that is understood, mute it and inspect which features the\npenalized model kept.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import skore\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.preprocessing import StandardScaler\n\nmodel_l1 = make_pipeline(\n StandardScaler(),\n LogisticRegression(\n l1_ratio=1.0,\n solver=\"saga\",\n max_iter=5_000,\n random_state=42,\n ),\n)\n\nreport_l1 = evaluate(\n model_l1,\n X=X,\n y=y,\n splitter=splitter,\n pos_label=pos_label,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SKD008 still fires on the correlated inputs.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "report_l1.checks.summarize(fast_mode=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Mute the expected tip and look at the fitted coefficients: many correlated\nfeatures are driven to zero.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "with skore.configuration(ignore_checks=[\"SKD008\"]):\n muted = report_l1.checks.summarize(fast_mode=True)\nmuted" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "_ = report_l1.inspection.coefficients().plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n\nSKD008 highlights redundant numeric features that may cause the fitting to\nfail or complicate interpretation. Here, the Associations view and Spearman\nclustering guided dropping and selecting cluster representatives; L1\n``LogisticRegression`` can shrink coefficients but does not clear the check,\nso mute SKD008 once that behavior is expected. Choose dropping or\ncluster-based selection based on how you want to modify the feature table.\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.7" } }, "nbformat": 4, "nbformat_minor": 0 }