.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/pitfalls_and_solutions/plot_skd004_high_class_imbalance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_pitfalls_and_solutions_plot_skd004_high_class_imbalance.py: .. _example_skd004_high_class_imbalance: SKD004 - High class imbalance ============================= :ref:`SKD004 ` flags a binary classification task when the majority class exceeds 80 % of rows. Accuracy can look high while the minority class is ignored as a default. This notebook is mostly about how to work with that imbalance once the check fires: we do not try to make SKD004 disappear, because natural prevalence is often the right thing to keep. What we do instead (see also :ref:`automated_checks`): - report absolute counts as well as percentages, - evaluate ranking and calibration (ROC AUC, log-loss) before trusting thresholded precision / recall, - tune the decision threshold under an explicit precision / recall or cost constraint (for example with :class:`~sklearn.model_selection.TunedThresholdClassifierCV`), - avoid ``class_weight`` and resampling when calibrated probabilities matter, - correct for prevalence shift if you collect minority-only data. See also: https://probabl-ai.github.io/calibration-cost-sensitive-learning/content/notebooks/imbalanced_classification.html We use Covertype forest types 2 (majority) vs 5 (minority) on an 8,000-row stratified subsample. The goal is to keep natural prevalence, judge probability quality first, then choose a cut-off that matches the precision / recall trade-off you care about. .. GENERATED FROM PYTHON SOURCE LINES 35-42 Load Covertype (types 2 vs 5) ============================= Types 2 vs 5 give a natural imbalance (type 2 is the majority class). We keep minority type 5 as the positive class and draw an 8,000-row stratified subsample so the gallery stays fast while absolute minority counts remain large enough to learn from. .. GENERATED FROM PYTHON SOURCE LINES 42-60 .. code-block:: Python import numpy as np from sklearn.datasets import fetch_covtype from sklearn.model_selection import train_test_split df = fetch_covtype(as_frame=True).frame pair = df.query("Cover_Type.isin([2, 5])") y_full = (pair["Cover_Type"] == 5).astype(int).rename("is_type_5") X_full = pair.drop(columns=["Cover_Type"]) X, _, y, _ = train_test_split( X_full, y_full, train_size=8_000, stratify=y_full, random_state=42, ) .. GENERATED FROM PYTHON SOURCE LINES 61-63 .. code-block:: Python y.value_counts(normalize=True).round(4) .. rst-class:: sphx-glr-script-out .. code-block:: none is_type_5 0 0.9676 1 0.0324 Name: proportion, dtype: float64 .. GENERATED FROM PYTHON SOURCE LINES 64-66 .. code-block:: Python y.value_counts() .. rst-class:: sphx-glr-script-out .. code-block:: none is_type_5 0 7741 1 259 Name: count, dtype: int64 .. GENERATED FROM PYTHON SOURCE LINES 67-68 Inspect the feature matrix with :class:`~skrub.TableReport`. .. GENERATED FROM PYTHON SOURCE LINES 68-73 .. code-block:: Python from skrub import TableReport TableReport(X) .. raw:: html

Column	Column name	dtype	Is sorted	Unique values	Mean	Std	Min	Median	Max
0	Elevation	Float64DType	False	906 (11.3%)	2.91e+03	187.	2.15e+03	2.93e+03	3.40e+03
1	Aspect	Float64DType	False	361 (4.5%)	151.	107.	0.00	126.	360.
2	Slope	Float64DType	False	44 (0.5%)	13.7	7.06	0.00	13.0	45.0
3	Horizontal_Distance_To_Hydrology	Float64DType	False	341 (4.3%)	276.	206.	0.00	240.	1.37e+03
4	Vertical_Distance_To_Hydrology	Float64DType	False	383 (4.8%)	45.5	56.4	-153.	30.0	549.
5	Horizontal_Distance_To_Roadways	Float64DType	False	3199 (40.0%)	2.36e+03	1.60e+03	30.0	1.97e+03	6.94e+03
6	Hillshade_9am	Float64DType	False	152 (1.9%)	214.	24.7	84.0	220.	254.
7	Hillshade_Noon	Float64DType	False	114 (1.4%)	225.	18.6	124.	227.	254.
8	Hillshade_3pm	Float64DType	False	231 (2.9%)	142.	36.5	0.00	142.	251.
9	Horizontal_Distance_To_Fire_Points	Float64DType	False	2841 (35.5%)	2.14e+03	1.40e+03	30.0	1.83e+03	7.08e+03
10	Wilderness_Area_0	Float64DType	False	2 (< 0.1%)	0.504	0.500	0.00	1.00	1.00
11	Wilderness_Area_1	Float64DType	False	2 (< 0.1%)	0.0319	0.176	0.00	0.00	1.00
12	Wilderness_Area_2	Float64DType	False	2 (< 0.1%)	0.451	0.498	0.00	0.00	1.00
13	Wilderness_Area_3	Float64DType	False	2 (< 0.1%)	0.0123	0.110	0.00	0.00	1.00
14	Soil_Type_0	Float64DType	True	1 (< 0.1%)	0.00	0.00
15	Soil_Type_1	Float64DType	False	2 (< 0.1%)	0.00500	0.0705	0.00	0.00	1.00
16	Soil_Type_2	Float64DType	False	2 (< 0.1%)	0.00413	0.0641	0.00	0.00	1.00
17	Soil_Type_3	Float64DType	False	2 (< 0.1%)	0.0140	0.117	0.00	0.00	1.00
18	Soil_Type_4	Float64DType	True	1 (< 0.1%)	0.00	0.00
19	Soil_Type_5	Float64DType	False	2 (< 0.1%)	0.00387	0.0621	0.00	0.00	1.00
20	Soil_Type_6	Float64DType	False	2 (< 0.1%)	0.000375	0.0194	0.00	0.00	1.00
21	Soil_Type_7	Float64DType	False	2 (< 0.1%)	0.000375	0.0194	0.00	0.00	1.00
22	Soil_Type_8	Float64DType	False	2 (< 0.1%)	0.00275	0.0524	0.00	0.00	1.00
23	Soil_Type_9	Float64DType	False	2 (< 0.1%)	0.0413	0.199	0.00	0.00	1.00
24	Soil_Type_10	Float64DType	False	2 (< 0.1%)	0.0318	0.175	0.00	0.00	1.00
25	Soil_Type_11	Float64DType	False	2 (< 0.1%)	0.0909	0.287	0.00	0.00	1.00
26	Soil_Type_12	Float64DType	False	2 (< 0.1%)	0.0480	0.214	0.00	0.00	1.00
27	Soil_Type_13	Float64DType	True	1 (< 0.1%)	0.00	0.00
28	Soil_Type_14	Float64DType	True	1 (< 0.1%)	0.00	0.00
29	Soil_Type_15	Float64DType	False	2 (< 0.1%)	0.00550	0.0740	0.00	0.00	1.00
30	Soil_Type_16	Float64DType	False	2 (< 0.1%)	0.00537	0.0731	0.00	0.00	1.00
31	Soil_Type_17	Float64DType	False	2 (< 0.1%)	0.00675	0.0819	0.00	0.00	1.00
32	Soil_Type_18	Float64DType	False	2 (< 0.1%)	0.00500	0.0705	0.00	0.00	1.00
33	Soil_Type_19	Float64DType	False	2 (< 0.1%)	0.0181	0.133	0.00	0.00	1.00
34	Soil_Type_20	Float64DType	True	1 (< 0.1%)	0.00	0.00
35	Soil_Type_21	Float64DType	False	2 (< 0.1%)	0.0238	0.152	0.00	0.00	1.00
36	Soil_Type_22	Float64DType	False	2 (< 0.1%)	0.0729	0.260	0.00	0.00	1.00
37	Soil_Type_23	Float64DType	False	2 (< 0.1%)	0.0338	0.181	0.00	0.00	1.00
38	Soil_Type_24	Float64DType	False	2 (< 0.1%)	0.00125	0.0353	0.00	0.00	1.00
39	Soil_Type_25	Float64DType	False	2 (< 0.1%)	0.00750	0.0863	0.00	0.00	1.00
40	Soil_Type_26	Float64DType	False	2 (< 0.1%)	0.00112	0.0335	0.00	0.00	1.00
41	Soil_Type_27	Float64DType	False	2 (< 0.1%)	0.00350	0.0591	0.00	0.00	1.00
42	Soil_Type_28	Float64DType	False	2 (< 0.1%)	0.250	0.433	0.00	0.00	1.00
43	Soil_Type_29	Float64DType	False	2 (< 0.1%)	0.0744	0.262	0.00	0.00	1.00
44	Soil_Type_30	Float64DType	False	2 (< 0.1%)	0.0467	0.211	0.00	0.00	1.00
45	Soil_Type_31	Float64DType	False	2 (< 0.1%)	0.105	0.307	0.00	0.00	1.00
46	Soil_Type_32	Float64DType	False	2 (< 0.1%)	0.0882	0.284	0.00	0.00	1.00
47	Soil_Type_33	Float64DType	False	2 (< 0.1%)	0.00387	0.0621	0.00	0.00	1.00
48	Soil_Type_34	Float64DType	True	1 (< 0.1%)	0.00	0.00
49	Soil_Type_35	Float64DType	True	1 (< 0.1%)	0.00	0.00
50	Soil_Type_36	Float64DType	True	1 (< 0.1%)	0.00	0.00
51	Soil_Type_37	Float64DType	False	2 (< 0.1%)	0.00250	0.0499	0.00	0.00	1.00
52	Soil_Type_38	Float64DType	False	2 (< 0.1%)	0.000875	0.0296	0.00	0.00	1.00
53	Soil_Type_39	Float64DType	False	2 (< 0.1%)	0.00112	0.0335	0.00	0.00	1.00

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

.. GENERATED FROM PYTHON SOURCE LINES 74-76 The binary target marks type-5 stands; the majority class exceeds 80 % of rows, so SKD004 will fire. .. GENERATED FROM PYTHON SOURCE LINES 76-79 .. code-block:: Python TableReport(y) .. raw:: html

	is_type_5
213,147	0
109,197	0
171,741	0
361,152	0
164,253	0

58,342	0
11,757	1
417,010	0
258,803	0
304,052	0

Column	Column name	dtype	Is sorted	Null values	Unique values	Mean	Std	Min	Median	Max
0	is_type_5	Int64DType	False	0 (0.0%)	2 (< 0.1%)	0.0324	0.177	0	0	1

Please enable javascript

.. GENERATED FROM PYTHON SOURCE LINES 80-86 Trigger SKD004 - default classifier on imbalanced labels ======================================================== A default gradient boosting classifier does not change label counts. The check cares about the class mix in the data, not about whether we reweighted the loss. .. GENERATED FROM PYTHON SOURCE LINES 86-102 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingClassifier from skore import TrainTestSplit, evaluate splitter = TrainTestSplit(test_size=0.2, random_state=42, stratify=y) classifier = HistGradientBoostingClassifier(random_state=42) report = evaluate( classifier, X=X, y=y, pos_label=1, splitter=splitter, ) report .. raw:: html

Metric	HistGradientBoostingClassifier
Accuracy	0.970625
Precision	0.608696
Recall	0.269231
ROC AUC	0.909586
Log loss	0.112095
Brier score	0.023782
Fit time (s)	0.530783
Predict time (s)	0.011547

HistGradientBoostingClassifier(random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

HistGradientBoostingClassifier

?Documentation for HistGradientBoostingClassifieriFitted

Parameters

	random_state random_state: int, RandomState instance or None, default=None Pseudo-random number generator to control the subsampling in the binning process, and the train/validation data split if early stopping is enabled. Pass an int for reproducible output across multiple function calls. See :term:`Glossary <random_state>`.	42
	loss loss: {'log_loss'}, default='log_loss' The loss function to use in the boosting process. For binary classification problems, 'log_loss' is also known as logistic loss, binomial deviance or binary crossentropy. Internally, the model fits one tree per boosting iteration and uses the logistic sigmoid function (expit) as inverse link function to compute the predicted positive class probability. For multiclass classification problems, 'log_loss' is also known as multinomial deviance or categorical crossentropy. Internally, the model fits one tree per boosting iteration and per class and uses the softmax function as inverse link function to compute the predicted probabilities of the classes.	'log_loss'
	learning_rate learning_rate: float, default=0.1 The learning rate, also known as shrinkage. This is used as a multiplicative factor for the leaves values. Use ``1`` for no shrinkage.	0.1
	max_iter max_iter: int, default=100 The maximum number of iterations of the boosting process, i.e. the maximum number of trees for binary classification. For multiclass classification, `n_classes` trees per iteration are built.	100
	max_leaf_nodes max_leaf_nodes: int or None, default=31 The maximum number of leaves for each tree. Must be strictly greater than 1. If None, there is no maximum limit.	31
	max_depth max_depth: int or None, default=None The maximum depth of each tree. The depth of a tree is the number of edges to go from the root to the deepest leaf. Depth isn't constrained by default.	None
	min_samples_leaf min_samples_leaf: int, default=20 The minimum number of samples per leaf. For small datasets with less than a few hundred samples, it is recommended to lower this value since only very shallow trees would be built.	20
	l2_regularization l2_regularization: float, default=0 The L2 regularization parameter penalizing leaves with small hessians. Use ``0`` for no regularization (default).	0.0
	max_features max_features: float, default=1.0 Proportion of randomly chosen features in each and every node split. This is a form of regularization, smaller values make the trees weaker learners and might prevent overfitting. If interaction constraints from `interaction_cst` are present, only allowed features are taken into account for the subsampling. .. versionadded:: 1.4	1.0
	max_bins max_bins: int, default=255 The maximum number of bins to use for non-missing values. Before training, each feature of the input array `X` is binned into integer-valued bins, which allows for a much faster training stage. Features with a small number of unique values may use less than ``max_bins`` bins. In addition to the ``max_bins`` bins, one more bin is always reserved for missing values. Must be no larger than 255.	255
	categorical_features categorical_features: array-like of {bool, int, str} of shape (n_features) or shape (n_categorical_features,), default='from_dtype' Indicates the categorical features. - None : no feature will be considered categorical. - boolean array-like : boolean mask indicating categorical features. - integer array-like : integer indices indicating categorical features. - str array-like: names of categorical features (assuming the training data has feature names). - `"from_dtype"`: dataframe columns with dtype "Categorical" and "Enum" are considered to be categorical features. The input must be a dataframe that is supported by narwhals (or supports it): :func:`narwhals.from_native` must work. This is the case, for instance, for pandas and polars DataFrames. For each categorical feature, there must be at most `max_bins` unique categories. Negative values for categorical features encoded as numeric dtypes are treated as missing values. All categorical values are converted to floating point numbers. This means that categorical values of 1.0 and 1 are treated as the same category. Read more in the :ref:`User Guide <categorical_support_gbdt>`. .. versionadded:: 0.24 .. versionchanged:: 1.2 Added support for feature names. .. versionchanged:: 1.4 Added `"from_dtype"` option. .. versionchanged:: 1.6 The default value changed from `None` to `"from_dtype"`.	'from_dtype'
	monotonic_cst monotonic_cst: array-like of int of shape (n_features) or dict, default=None Monotonic constraint to enforce on each feature are specified using the following integer values: - 1: monotonic increase - 0: no constraint - -1: monotonic decrease If a dict with str keys, map feature to monotonic constraints by name. If an array, the features are mapped to constraints by position. See :ref:`monotonic_cst_features_names` for a usage example. The constraints are only valid for binary classifications and hold over the probability of the positive class. Read more in the :ref:`User Guide <monotonic_cst_gbdt>`. .. versionadded:: 0.23 .. versionchanged:: 1.2 Accept dict of constraints with feature names as keys.	None
	interaction_cst interaction_cst: {"pairwise", "no_interactions"} or sequence of lists/tuples/sets of int, default=None Specify interaction constraints, the sets of features which can interact with each other in child node splits. Each item specifies the set of feature indices that are allowed to interact with each other. If there are more features than specified in these constraints, they are treated as if they were specified as an additional set. The strings "pairwise" and "no_interactions" are shorthands for allowing only pairwise or no interactions, respectively. For instance, with 5 features in total, `interaction_cst=[{0, 1}]` is equivalent to `interaction_cst=[{0, 1}, {2, 3, 4}]`, and specifies that each branch of a tree will either only split on features 0 and 1 or only split on features 2, 3 and 4. See :ref:`this example<ice-vs-pdp>` on how to use `interaction_cst`. .. versionadded:: 1.2	None
	warm_start warm_start: bool, default=False When set to ``True``, reuse the solution of the previous call to fit and add more estimators to the ensemble. For results to be valid, the estimator should be re-trained on the same data only. See :term:`the Glossary <warm_start>`.	False
	early_stopping early_stopping: 'auto' or bool, default='auto' If 'auto', early stopping is enabled if the sample size is larger than 10000 or if `X_val` and `y_val` are passed to `fit`. If True, early stopping is enabled, otherwise early stopping is disabled. .. versionadded:: 0.23	'auto'
	scoring scoring: str or callable or None, default='loss' Scoring method to use for early stopping. Only used if `early_stopping` is enabled. Options: - str: see :ref:`scoring_string_names` for options. - callable: a scorer callable object (e.g., function) with signature ``scorer(estimator, X, y)``. See :ref:`scoring_callable` for details. - `None`: :ref:`accuracy <accuracy_score>` is used. - 'loss': early stopping is checked w.r.t the loss value.	'loss'
	validation_fraction validation_fraction: int or float or None, default=0.1 Proportion (or absolute size) of training data to set aside as validation data for early stopping. If None, early stopping is done on the training data. The value is ignored if either early stopping is not performed, e.g. `early_stopping=False`, or if `X_val` and `y_val` are passed to fit.	0.1
	n_iter_no_change n_iter_no_change: int, default=10 Used to determine when to "early stop". The fitting process is stopped when none of the last ``n_iter_no_change`` scores are better than the ``n_iter_no_change - 1`` -th-to-last one, up to some tolerance. Only used if early stopping is performed.	10
	tol tol: float, default=1e-7 The absolute tolerance to use when comparing scores. The higher the tolerance, the more likely we are to early stop: higher tolerance means that it will be harder for subsequent iterations to be considered an improvement upon the reference score.	1e-07
	verbose verbose: int, default=0 The verbosity level. If not zero, print some information about the fitting process. ``1`` prints only summary info, ``2`` prints info per iteration.	0
	class_weight class_weight: dict or 'balanced', default=None Weights associated with classes in the form `{class_label: weight}`. If not given, all classes are supposed to have weight one. The "balanced" mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as `n_samples / (n_classes * np.bincount(y))`. Note that these weights will be multiplied with sample_weight (passed through the fit method) if `sample_weight` is specified. .. versionadded:: 1.2	None

Fitted attributes

Name	Type	Value
classes_ classes_: array, shape = (n_classes,) Class labels.	ndarray[int64](2,)	[0,1]
do_early_stopping_ do_early_stopping_: bool Indicates whether early stopping is used during training.	bool	False
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0	ndarray[object](54,)	['Elevation','Aspect','Slope',...,'Soil_Type_37','Soil_Type_38', 'Soil_Type_39']
is_categorical_ is_categorical_: ndarray, shape (n_features, ) or None Boolean mask for the categorical features. ``None`` if there are no categorical features.	NoneType	None
n_features_in_ n_features_in_: int Number of features seen during :term:`fit`. .. versionadded:: 0.24	int	54
n_iter_ n_iter_: int The number of iterations as selected by early stopping, depending on the `early_stopping` parameter. Otherwise it corresponds to max_iter.	int	100
n_trees_per_iteration_ n_trees_per_iteration_: int The number of tree that are built at each iteration. This is equal to 1 for binary classification, and to ``n_classes`` for multiclass classification.	int	1
train_score_ train_score_: ndarray, shape (n_iter_+1,) The scores at each iteration on the training data. The first entry is the score of the ensemble before the first iteration. Scores are computed according to the ``scoring`` parameter. If ``scoring`` is not 'loss', scores are computed on a subset of at most 10 000 samples. Empty if no early stopping.	ndarray[float64](0,)	[]
validation_score_ validation_score_: ndarray, shape (n_iter_+1,) The scores at each iteration on the held-out validation data. The first entry is the score of the ensemble before the first iteration. Scores are computed according to the ``scoring`` parameter. Empty if no early stopping or if ``validation_fraction`` is None.	ndarray[float64](0,)	[]

Column	Column name	dtype	Is sorted	Unique values	Mean	Std	Min	Median	Max
0	Elevation	Float64DType	False	906 (11.3%)	2.91e+03	187.	2.15e+03	2.93e+03	3.40e+03
1	Aspect	Float64DType	False	361 (4.5%)	151.	107.	0.00	126.	360.
2	Slope	Float64DType	False	44 (0.5%)	13.7	7.06	0.00	13.0	45.0
3	Horizontal_Distance_To_Hydrology	Float64DType	False	341 (4.3%)	276.	206.	0.00	240.	1.37e+03
4	Vertical_Distance_To_Hydrology	Float64DType	False	383 (4.8%)	45.5	56.4	-153.	30.0	549.
5	Horizontal_Distance_To_Roadways	Float64DType	False	3199 (40.0%)	2.36e+03	1.60e+03	30.0	1.97e+03	6.94e+03
6	Hillshade_9am	Float64DType	False	152 (1.9%)	214.	24.7	84.0	220.	254.
7	Hillshade_Noon	Float64DType	False	114 (1.4%)	225.	18.6	124.	227.	254.
8	Hillshade_3pm	Float64DType	False	231 (2.9%)	142.	36.5	0.00	142.	251.
9	Horizontal_Distance_To_Fire_Points	Float64DType	False	2841 (35.5%)	2.14e+03	1.40e+03	30.0	1.83e+03	7.08e+03
10	Wilderness_Area_0	Float64DType	False	2 (< 0.1%)	0.504	0.500	0.00	1.00	1.00
11	Wilderness_Area_1	Float64DType	False	2 (< 0.1%)	0.0319	0.176	0.00	0.00	1.00
12	Wilderness_Area_2	Float64DType	False	2 (< 0.1%)	0.451	0.498	0.00	0.00	1.00
13	Wilderness_Area_3	Float64DType	False	2 (< 0.1%)	0.0123	0.110	0.00	0.00	1.00
14	Soil_Type_0	Float64DType	True	1 (< 0.1%)	0.00	0.00
15	Soil_Type_1	Float64DType	False	2 (< 0.1%)	0.00500	0.0705	0.00	0.00	1.00
16	Soil_Type_2	Float64DType	False	2 (< 0.1%)	0.00413	0.0641	0.00	0.00	1.00
17	Soil_Type_3	Float64DType	False	2 (< 0.1%)	0.0140	0.117	0.00	0.00	1.00
18	Soil_Type_4	Float64DType	True	1 (< 0.1%)	0.00	0.00
19	Soil_Type_5	Float64DType	False	2 (< 0.1%)	0.00387	0.0621	0.00	0.00	1.00
20	Soil_Type_6	Float64DType	False	2 (< 0.1%)	0.000375	0.0194	0.00	0.00	1.00
21	Soil_Type_7	Float64DType	False	2 (< 0.1%)	0.000375	0.0194	0.00	0.00	1.00
22	Soil_Type_8	Float64DType	False	2 (< 0.1%)	0.00275	0.0524	0.00	0.00	1.00
23	Soil_Type_9	Float64DType	False	2 (< 0.1%)	0.0413	0.199	0.00	0.00	1.00
24	Soil_Type_10	Float64DType	False	2 (< 0.1%)	0.0318	0.175	0.00	0.00	1.00
25	Soil_Type_11	Float64DType	False	2 (< 0.1%)	0.0909	0.287	0.00	0.00	1.00
26	Soil_Type_12	Float64DType	False	2 (< 0.1%)	0.0480	0.214	0.00	0.00	1.00
27	Soil_Type_13	Float64DType	True	1 (< 0.1%)	0.00	0.00
28	Soil_Type_14	Float64DType	True	1 (< 0.1%)	0.00	0.00
29	Soil_Type_15	Float64DType	False	2 (< 0.1%)	0.00550	0.0740	0.00	0.00	1.00
30	Soil_Type_16	Float64DType	False	2 (< 0.1%)	0.00537	0.0731	0.00	0.00	1.00
31	Soil_Type_17	Float64DType	False	2 (< 0.1%)	0.00675	0.0819	0.00	0.00	1.00
32	Soil_Type_18	Float64DType	False	2 (< 0.1%)	0.00500	0.0705	0.00	0.00	1.00
33	Soil_Type_19	Float64DType	False	2 (< 0.1%)	0.0181	0.133	0.00	0.00	1.00
34	Soil_Type_20	Float64DType	True	1 (< 0.1%)	0.00	0.00
35	Soil_Type_21	Float64DType	False	2 (< 0.1%)	0.0238	0.152	0.00	0.00	1.00
36	Soil_Type_22	Float64DType	False	2 (< 0.1%)	0.0729	0.260	0.00	0.00	1.00
37	Soil_Type_23	Float64DType	False	2 (< 0.1%)	0.0338	0.181	0.00	0.00	1.00
38	Soil_Type_24	Float64DType	False	2 (< 0.1%)	0.00125	0.0353	0.00	0.00	1.00
39	Soil_Type_25	Float64DType	False	2 (< 0.1%)	0.00750	0.0863	0.00	0.00	1.00
40	Soil_Type_26	Float64DType	False	2 (< 0.1%)	0.00112	0.0335	0.00	0.00	1.00
41	Soil_Type_27	Float64DType	False	2 (< 0.1%)	0.00350	0.0591	0.00	0.00	1.00
42	Soil_Type_28	Float64DType	False	2 (< 0.1%)	0.250	0.433	0.00	0.00	1.00
43	Soil_Type_29	Float64DType	False	2 (< 0.1%)	0.0744	0.262	0.00	0.00	1.00
44	Soil_Type_30	Float64DType	False	2 (< 0.1%)	0.0467	0.211	0.00	0.00	1.00
45	Soil_Type_31	Float64DType	False	2 (< 0.1%)	0.105	0.307	0.00	0.00	1.00
46	Soil_Type_32	Float64DType	False	2 (< 0.1%)	0.0882	0.284	0.00	0.00	1.00
47	Soil_Type_33	Float64DType	False	2 (< 0.1%)	0.00387	0.0621	0.00	0.00	1.00
48	Soil_Type_34	Float64DType	True	1 (< 0.1%)	0.00	0.00
49	Soil_Type_35	Float64DType	True	1 (< 0.1%)	0.00	0.00
50	Soil_Type_36	Float64DType	True	1 (< 0.1%)	0.00	0.00
51	Soil_Type_37	Float64DType	False	2 (< 0.1%)	0.00250	0.0499	0.00	0.00	1.00
52	Soil_Type_38	Float64DType	False	2 (< 0.1%)	0.000875	0.0296	0.00	0.00	1.00
53	Soil_Type_39	Float64DType	False	2 (< 0.1%)	0.00112	0.0335	0.00	0.00	1.00
54	is_type_5	Int64DType	False	2 (< 0.1%)	0.0324	0.177	0	0	1

Please enable javascript

.. GENERATED FROM PYTHON SOURCE LINES 103-105 SKD004 should fire: the majority class exceeds 80 % of rows. Ignore SKD008 to avoid correlated-feature warnings from Covertype's constant soil one-hots. .. GENERATED FROM PYTHON SOURCE LINES 105-108 .. code-block:: Python report.checks.summarize(fast_mode=True, ignore=["SKD008"]) .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 109-117 Accuracy and F1 are poor defaults under imbalance ================================================= Accuracy can be inflated by nearly always predicting the majority class. F1 averages precision and recall into one number and hides which side of the trade-off you care about, so we do not use it here. At the default probability cut-off of 0.5, minority recall is often weak because rare events receive small predicted probabilities. .. GENERATED FROM PYTHON SOURCE LINES 117-123 .. code-block:: Python report.metrics.summarize( metric=["accuracy", "precision", "recall", "roc_auc", "log_loss"], data_source="both", ).frame() .. raw:: html

	HistGradientBoostingClassifier (train)	HistGradientBoostingClassifier (test)
metric
accuracy	1.000000	0.970625
precision	1.000000	0.608696
recall	1.000000	0.269231
roc_auc	1.000000	0.909586
log_loss	0.005539	0.112095

.. GENERATED FROM PYTHON SOURCE LINES 124-127 The stratified hold-out still has only a few dozen type-5 rows against about 1,500 majority rows. Most of those scarce positives are predicted as majority, so minority recall is low while accuracy stays high. .. GENERATED FROM PYTHON SOURCE LINES 127-130 .. code-block:: Python report.metrics.confusion_matrix().plot() .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_001.png :alt: Confusion Matrix Data source: Test set :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none

.. GENERATED FROM PYTHON SOURCE LINES 131-154 Check ranking and calibration first =================================== Before touching thresholds, ask whether probabilities are any good: - ROC AUC asks whether positives tend to get higher scores than negatives (threshold-free ranking), - log-loss penalizes confident wrong probabilities, - a calibration curve asks whether predicted probabilities match observed frequencies. On the calibration plot, bins of predicted probability are compared to the fraction of true positives in each bin. A useful curve hugs the diagonal: when the model says "20 %", about 20 % of those rows really are positive. Points above the diagonal mean under-confidence (events happen more often than predicted); points below mean over-confidence (the model is too sure). With a rare class, almost all mass sits at low probabilities, so the curve often only appears on the left of the plot; that is expected, not a plotting bug. If ranking and calibration look reasonable, the model may already be useful; the default 0.5 cut-off is simply the wrong operating point for a rare class. The next section shows how ``class_weight="balanced"`` can push the curve below the diagonal by inflating minority probabilities. .. GENERATED FROM PYTHON SOURCE LINES 154-160 .. code-block:: Python report.metrics.summarize( metric=["roc_auc", "log_loss"], data_source="test", ).frame() .. rst-class:: sphx-glr-script-out .. code-block:: none metric roc_auc 0.909586 log_loss 0.112095 Name: HistGradientBoostingClassifier, dtype: float64 .. GENERATED FROM PYTHON SOURCE LINES 161-163 .. code-block:: Python report.inspection.calibration_curve(data_source="test", n_bins=10).plot() .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_002.png :alt: Calibration Curve of HistGradientBoostingClassifier Positive label: 1 :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none

.. GENERATED FROM PYTHON SOURCE LINES 164-177 Class weights as a cautionary comparison ======================================== A common reflex is ``class_weight="balanced"``. Rebalancing with weights is equivalent in spirit to resampling methods such as SMOTE or random oversampling / undersampling: they change the effective class mix and will suffer from the same issues. That often improves precision / recall at 0.5 because it inflates minority probabilities, but it typically breaks calibration: predicted probabilities run ahead of observed rates, so the curve drifts below the diagonal (over-confidence on the originally rare class). If you later recalibrate, the thresholded gains often disappear. We show the comparison, then leave weights behind when calibrated probabilities matter. .. GENERATED FROM PYTHON SOURCE LINES 177-196 .. code-block:: Python from skore import compare report_weighted = evaluate( HistGradientBoostingClassifier(class_weight="balanced", random_state=42), X=X, y=y, pos_label=1, splitter=splitter, ) comparison_weights = compare( {"default": report, "class_weight_balanced": report_weighted} ) comparison_weights.metrics.summarize( metric=["precision", "recall", "roc_auc", "log_loss"], data_source="test", ).frame() .. raw:: html

estimator	default	class_weight_balanced
metric
precision	0.608696	0.428571
recall	0.269231	0.461538
roc_auc	0.909586	0.911225
log_loss	0.112095	0.111834

.. GENERATED FROM PYTHON SOURCE LINES 197-199 After reweighting, compare this curve to the default one: points tend to sit further below the diagonal (over-confident on the rare class). .. GENERATED FROM PYTHON SOURCE LINES 199-202 .. code-block:: Python report_weighted.inspection.calibration_curve(data_source="test", n_bins=10).plot() .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_003.png :alt: Calibration Curve of HistGradientBoostingClassifier Positive label: 1 :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_003.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none

.. GENERATED FROM PYTHON SOURCE LINES 203-216 Tune the decision threshold =========================== Keep the default, prevalence-correct model and change only the decision rule. For this demo we require at least 30 % precision on type 5, then maximize recall. That floor is an explicit product choice: high enough to limit false alarms, low enough that a rare-event model can still catch a useful share of true type-5 stands. Replace 0.3 with a cost or capacity constraint in real work. :class:`~sklearn.model_selection.TunedThresholdClassifierCV` searches the cut-off by cross-validation and does not change ``predict_proba``, so calibration stays intact. .. GENERATED FROM PYTHON SOURCE LINES 216-247 .. code-block:: Python from sklearn.metrics import make_scorer, precision_score, recall_score from sklearn.model_selection import TunedThresholdClassifierCV def recall_with_min_precision(y_true, y_pred, precision_level=0.3): """Maximize recall only among thresholds that keep precision high enough.""" precision = precision_score(y_true, y_pred, zero_division=0) recall = recall_score(y_true, y_pred, zero_division=0) if precision < precision_level: return -np.inf return recall threshold_scoring = make_scorer(recall_with_min_precision, precision_level=0.3) tuned = TunedThresholdClassifierCV( estimator=HistGradientBoostingClassifier(random_state=42), scoring=threshold_scoring, cv=3, n_jobs=4, ) report_tuned = evaluate( tuned, X=X, y=y, pos_label=1, splitter=splitter, ) .. GENERATED FROM PYTHON SOURCE LINES 248-251 SKD004 still fires: label counts did not change. That is expected. We improved how we decide, not the histogram SKD004 reads. Ignore SKD008 to avoid correlated-feature warnings from Covertype's constant soil one-hots. .. GENERATED FROM PYTHON SOURCE LINES 251-254 .. code-block:: Python report_tuned.checks.summarize(fast_mode=True, ignore=["SKD008"]) .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 255-262 .. code-block:: Python print("Chosen decision threshold:", float(report_tuned.estimator_.best_threshold_)) report_tuned.metrics.summarize( metric=["accuracy", "precision", "recall", "roc_auc", "log_loss"], data_source="both", ).frame() .. rst-class:: sphx-glr-script-out .. code-block:: none Chosen decision threshold: 0.020068079655242547 .. raw:: html

	TunedThresholdClassifierCV (train)	TunedThresholdClassifierCV (test)
metric
accuracy	0.966094	0.921875
precision	0.488208	0.241135
recall	1.000000	0.653846
roc_auc	1.000000	0.909586
log_loss	0.005539	0.112095

.. GENERATED FROM PYTHON SOURCE LINES 263-265 .. code-block:: Python report_tuned.metrics.confusion_matrix().plot() .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_004.png :alt: Confusion Matrix Data source: Test set :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_004.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none

.. GENERATED FROM PYTHON SOURCE LINES 266-275 Compare the default 0.5 cut-off to the tuned threshold. Same underlying probabilities; only the hard predictions change. Precision / recall move; ROC AUC and log-loss stay essentially the same. The scorer asks for precision of at least 0.3, then maximizes recall: catch more type-5 stands without too many false alarms (fraud review, maintenance tickets, medical triage). Preferring high precision instead fits cases where a false alarm is costly: auto-blocking users, expensive tests, or limited outreach budgets. .. GENERATED FROM PYTHON SOURCE LINES 275-287 .. code-block:: Python comparison_thresholds = compare( { "default_threshold_0.5": report, "tuned_threshold": report_tuned, } ) comparison_thresholds.metrics.summarize( metric=["precision", "recall", "roc_auc", "log_loss"], data_source="test", ).frame() .. raw:: html

estimator	default_threshold_0.5	tuned_threshold
metric
precision	0.608696	0.241135
recall	0.269231	0.653846
roc_auc	0.909586	0.909586
log_loss	0.112095	0.112095

.. GENERATED FROM PYTHON SOURCE LINES 288-293 Inspect the precision-recall curve ================================== The dashed line is our precision floor (0.3). The tuned threshold should land near the highest-recall point that still respects that floor. .. GENERATED FROM PYTHON SOURCE LINES 293-310 .. code-block:: Python threshold = float(report_tuned.estimator_.best_threshold_) display = report.metrics.precision_recall() fig = display.plot() ax = fig.axes[0] ax.axhline(0.3, linestyle="--", color="gray", label="precision floor 0.3") ax.axvline( report_tuned.metrics.recall(), linestyle=":", color="C1", label=f"recall at threshold={threshold:.3f}", ) ax.legend(loc="best") ax.set_title("Precision-recall curve (test fold)") fig .. image-sg:: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_005.png :alt: Precision-Recall Curve for HistGradientBoostingClassifier Positive label: 1 Data source: Test set, Precision-recall curve (test fold) :srcset: /auto_examples/pitfalls_and_solutions/images/sphx_glr_plot_skd004_high_class_imbalance_005.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none

.. GENERATED FROM PYTHON SOURCE LINES 311-319 Collecting more minority data ============================= Gathering more type-5 plots can help the model see the rare class. If acquisition preferentially samples minority rows, train prevalence no longer matches production. Probabilities and thresholds fitted on that mix will be biased unless you correct for the shift. Clearing SKD004 by stuffing minority rows into the table is therefore not automatically a success. .. GENERATED FROM PYTHON SOURCE LINES 321-328 Conclusion ========== SKD004 warns that one class dominates the table. Keep natural prevalence when you need honest probabilities; move the threshold when you need a different precision / recall trade-off. Class weights and resampling are risky shortcuts if calibration matters for the decisions you deploy. .. rst-class:: sphx-glr-timing **Total running time of the script:** (1 minutes 16.986 seconds) .. _sphx_glr_download_auto_examples_pitfalls_and_solutions_plot_skd004_high_class_imbalance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_skd004_high_class_imbalance.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_skd004_high_class_imbalance.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_skd004_high_class_imbalance.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_

Elevation

Aspect

Slope

Horizontal_Distance_To_Hydrology

Vertical_Distance_To_Hydrology

Horizontal_Distance_To_Roadways

Hillshade_9am

Hillshade_Noon

Hillshade_3pm

Horizontal_Distance_To_Fire_Points

Wilderness_Area_0

Wilderness_Area_1

Wilderness_Area_2

Wilderness_Area_3

Soil_Type_0

Soil_Type_1

Soil_Type_2

Soil_Type_3

Soil_Type_4

Soil_Type_5

Soil_Type_6

Soil_Type_7

Soil_Type_8

Soil_Type_9

Soil_Type_10

Soil_Type_11

Soil_Type_12

Soil_Type_13

Soil_Type_14

Soil_Type_15

Soil_Type_16

Soil_Type_17

Soil_Type_18

Soil_Type_19

Soil_Type_20

Soil_Type_21

Soil_Type_22

Soil_Type_23

Soil_Type_24

Soil_Type_25

Soil_Type_26

Soil_Type_27

Soil_Type_28

Soil_Type_29

Soil_Type_30

Soil_Type_31

Soil_Type_32

Soil_Type_33

Soil_Type_34

Soil_Type_35

Soil_Type_36

Soil_Type_37

Soil_Type_38

Soil_Type_39

Please enable javascript

is_type_5

is_type_5

Please enable javascript

Elevation

Aspect

Slope

Horizontal_Distance_To_Hydrology

Vertical_Distance_To_Hydrology

Horizontal_Distance_To_Roadways

Hillshade_9am

Hillshade_Noon

Hillshade_3pm

Horizontal_Distance_To_Fire_Points

Wilderness_Area_0

Wilderness_Area_1

Wilderness_Area_2

Wilderness_Area_3

Soil_Type_0

Soil_Type_1

Soil_Type_2

Soil_Type_3

Soil_Type_4

Soil_Type_5

Soil_Type_6

Soil_Type_7