.. _sphx_glr_auto_examples_pitfalls_and_solutions:

Checking for modeling pitfalls and remediation
----------------------------------------------

Examples for each automated check. Each script reproduces a check, and walks
through fixes described in :ref:`automated_checks`.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="skd002-underfitting and skd001-overfitting describe opposite ends of the same problem: model expressiveness.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd001_skd002_overfitting_underfitting_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd001_skd002_overfitting_underfitting`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD001 & SKD002 - Overfitting and underfitting</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="skd003-inconsistent-performance flags folds whose test metrics diverge sharply from the median during a cross-validation evaluation. With a proper splitter this is often a diagnostic check: the data have structure (groups, time, or a corrupted batch) that shuffled cross-validation would hide.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd003_inconsistent_performance_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd003_inconsistent_performance`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD003 - Inconsistent performance across splits</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="skd004-high-class-imbalance flags a binary classification task when the majority class exceeds 80 % of rows. Accuracy can look high while the minority class is ignored as a default. This notebook is mostly about how to work with that imbalance once the check fires: we do not try to make SKD004 disappear, because natural prevalence is often the right thing to keep.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd004_high_class_imbalance_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd004_high_class_imbalance`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD004 - High class imbalance</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="skd005-underrepresented-classes flags a multiclass task when one or more classes each represent less than 10 % of rows. Overall accuracy can look acceptable while rare labels are barely learned. This notebook is about how to work with that rarity once the check fires: we do not try to make SKD005 disappear by reshaping the class histogram.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd005_underrepresented_classes_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd005_underrepresented_classes`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD005 - Underrepresented classes</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example demonstrates the limitations that skd007-mdi-cardinality-bias warns against on tree models. Mean decrease in impurity (MDI) tends to rank high-cardinality categorical or continuous columns as more important than features with as much signal but lower cardinality. This is due to the tree building process picking high cardinality features more often as they offer more split points to choose from.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd007_mdi_cardinality_bias_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd007_mdi_cardinality_bias`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD007 - MDI feature importance is biased for high-cardinality features</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example walks through mitigations when check skd008-correlated-features fires because numeric columns are nearly redundant. The check computes pairwise Spearman correlation on training inputs and flags pairs with |ρ| &gt; 0.9.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd008_correlated_features_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd008_correlated_features`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD008 - Highly correlated input features</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example demonstrates mitigations when check skd013-train-test-time-overlap fires on temporal data. The check compares datetime columns in train and test folds and flags overlap when the latest training timestamp is not strictly before the earliest test timestamp.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd013_train_test_time_overlap_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd013_train_test_time_overlap`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD013 - Train-test overlap in time series</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example walks through mitigations when checks skd014-hyperparams-at-search-edge and skd015-hyperparameters-worth-tuning fire on a fitted search object. SKD014 is an issue: numeric best_params_ sit on the minimum or maximum value tried, so the true optimum may lie outside the searched range. SKD015 is a tip: a hyperparameter is missing from the search space, which is incomplete rather than necessarily wrong.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd014_skd015_hyperparameter_search_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd014_skd015_hyperparameter_search`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD014 & SKD015: Hyperparameter search pitfalls</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example walks through mitigations when check skd016-estimator-not-tuned tips on a plain estimator left at scikit-learn defaults. The check compares initialization parameters against a curated table of high-impact hyperparameters and suggests axes worth tuning.">

.. only:: html

  .. image:: /auto_examples/pitfalls_and_solutions/images/thumb/sphx_glr_plot_skd016_estimator_not_tuned_thumb.png
    :alt:

  :doc:`/auto_examples/pitfalls_and_solutions/plot_skd016_estimator_not_tuned`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">SKD016 - Estimator not tuned</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /auto_examples/pitfalls_and_solutions/plot_skd001_skd002_overfitting_underfitting
   /auto_examples/pitfalls_and_solutions/plot_skd003_inconsistent_performance
   /auto_examples/pitfalls_and_solutions/plot_skd004_high_class_imbalance
   /auto_examples/pitfalls_and_solutions/plot_skd005_underrepresented_classes
   /auto_examples/pitfalls_and_solutions/plot_skd007_mdi_cardinality_bias
   /auto_examples/pitfalls_and_solutions/plot_skd008_correlated_features
   /auto_examples/pitfalls_and_solutions/plot_skd013_train_test_time_overlap
   /auto_examples/pitfalls_and_solutions/plot_skd014_skd015_hyperparameter_search
   /auto_examples/pitfalls_and_solutions/plot_skd016_estimator_not_tuned