# Cross-validation

> Rotate the validation fold k times and average, so every point is used for validation exactly once.

Source: https://www.mlfoundry.in/learn/cross-validation  
Section: ML Algorithms › Evaluation & Generalization  
Level: core · 19 min read · updated 2026-10-08

**k-fold cross-validation:** splits the data into $k$ disjoint folds, trains the model $k$ times (each time on $k-1$ folds) and scores it on the fold that was left out. The average of the $k$ held-out scores estimates how well **this modelling procedure** does on unseen data. Every sample is validated exactly once and used for training $k-1$ times.

- **Purpose:** Estimate generalization error; compare models; choose hyperparameters
- **Input → output:** data, a learning procedure and $k$ → $k$ fold scores, their mean and standard error
- **Cost:** $k$ model fits (LOOCV: $n$ fits, or **one** fit for linear smoothers)
- **Typical k:** 5 or 10; LOOCV (k = n) for very small data or cheap closed-form models
- **Variants:** stratified (class ratios), group (no group in two folds), time series (train on the past), repeated, nested
- **Assumes:** folds are exchangeable: each one looks like the future data, and nothing leaks between train and validation

## The problem

Training error is optimistic: the model has seen those points. A single held-out validation set fixes the bias but wastes data and is **noisy**: with 200 samples a 20% split scores the model on just 40 points, and a different random split can easily move accuracy by several points. On small or medium datasets we want an estimate that uses every point for validation *and* every point for training, without ever scoring a model on data it was fitted to.

Cross-validation does this by rotation. Cut the data into $k$ folds. In round $j$, hold out fold $j$, fit on the rest, score on fold $j$. After $k$ rounds every point has been predicted exactly once by a model that never saw it.

**k-fold cross-validation**

1. **Shuffle** the indices (unless order matters, e.g. time) and split them into $k$ folds $F_1,\dots,F_k$ of nearly equal size.
2. **For** $j=1\dots k$: fit the *whole pipeline* (scaling, feature selection, model) on $D\setminus F_j$.
3. **Score** that fitted model on $F_j$ only, giving the fold loss $L_j$.
4. **Report** the mean $\mathrm{CV}_k$ and its standard error. Then, for deployment, refit once on **all** the data.

## The estimate and its uncertainty

$$
L_j=\frac{1}{|F_j|}\sum_{i\in F_j}\ell\big(y_i,\ \hat f^{(-j)}(x_i)\big)\qquad \mathrm{CV}_k=\frac1k\sum_{j=1}^{k}L_j
$$

- $F_j$: indices in fold $j$
- $\hat f^{(-j)}$: model fitted on every fold except $F_j$
- $\ell$: per-sample loss (squared error, 0–1 loss, log-loss …)
- $L_j$: average loss on fold $j$
- $\mathrm{CV}_k$: the $k$-fold estimate of test error

With equal fold sizes this is the same as averaging the loss over all $n$ out-of-fold predictions. (scikit-learn's `cross_val_score` returns the $k$ fold scores, and you average them; with unequal folds the two averages differ slightly.)

$$
\widehat{\mathrm{SE}}=\frac{s_L}{\sqrt k},\qquad s_L^2=\frac{1}{k-1}\sum_{j=1}^{k}\big(L_j-\mathrm{CV}_k\big)^2
$$

- $s_L$: sample standard deviation of the k fold losses
- $\widehat{\mathrm{SE}}$: standard error of the fold mean: the usual error bar on a CV score

This formula treats the $L_j$ as independent. They are not. Any two training sets share $k-2$ of their $k$ folds, so the $k$ fitted models are strongly correlated, and with them the fold losses. Write $\sigma^2$ for the variance of one $L_j$ and $\rho$ for the correlation between two of them. Then

$$
\operatorname{Var}(\mathrm{CV}_k)=\frac{\sigma^2}{k}\big(1+(k-1)\rho\big)
$$

- $\sigma^2$: variance of a single fold loss
- $\rho$: average correlation between two fold losses (positive: overlapping training sets)

With $\rho>0$ the true variance is larger than $\sigma^2/k$, so $s_L/\sqrt k$ **under-states** the uncertainty. Bengio and Grandvalet (2004) showed that no unbiased estimator of this variance exists from a single run of k-fold CV. Read the SE as a rough, optimistic error bar. It is still useful (the "one-standard-error rule" picks the simplest model within one SE of the best), but two models whose CV scores differ by less than about one SE are effectively tied.

> **Which error does CV estimate?:** $\mathrm{CV}_k$ is computed from models trained on $n(k-1)/k$ points, not on the final model trained on $n$. So it estimates the expected error of *the procedure* at a slightly smaller sample size, averaged over training sets. It is a much less precise estimate of the error of the one model you finally ship.

## Choosing k: bias versus variance

Each fold model sees a fraction $(k-1)/k$ of the data. Learning curves usually still rise at that point, so fold models are slightly worse than the final model and $\mathrm{CV}_k$ is **pessimistically biased**. The bias is largest for small $k$: 2-fold trains on only half the data.

How k trades bias, variance and compute.

| k | Train size per fold | Bias (pessimism) | Variance of the estimate | Fits |
| --- | --- | --- | --- | --- |
| 2 | n/2 | high | moderate (few, noisy folds) | 2 |
| **5–10** | **80–90% of n** | **small** | **moderate** | **5–10** |
| n (LOOCV) | n − 1 | almost none | often high: the n models are nearly identical, so their errors are highly correlated | n (or 1) |

**LOOCV** is nearly unbiased, because each model trains on $n-1$ points. But its $n$ training sets differ by a single point, so the fold losses are almost perfectly correlated ($\rho\to1$ above), and its variance does not shrink the way $1/n$ would suggest. For unstable learners (deep trees, 1-NN) it can be quite noisy. Each fold also scores just one point, so with 0–1 loss every $L_j$ is 0 or 1. The empirical consensus (Breiman and Spector 1992; Kohavi 1995) is that **5- or 10-fold** gives the best trade-off. **Repeated** k-fold (e.g. 5 × 2 or 10 × 10 with different shuffles) averages away the luck of one particular partition.

## The LOOCV shortcut for linear smoothers

For any model whose fitted values are a fixed linear function of $y$, $\hat y = Hy$, you don't need to refit $n$ times. This covers least squares, ridge, polynomial and spline regression, and kernel smoothers with fixed bandwidth. Deleting point $i$ and refitting gives exactly:

$$
y_i-\hat f^{(-i)}(x_i)=\frac{y_i-\hat y_i}{1-h_{ii}}\qquad \mathrm{LOOCV}=\frac1n\sum_{i=1}^{n}\Big(\frac{y_i-\hat y_i}{1-h_{ii}}\Big)^2
$$

- $H$: hat (smoother) matrix; for OLS $H=X(X^\top X)^{-1}X^\top$, for ridge $X(X^\top X+\lambda I)^{-1}X^\top$
- $\hat y_i$: fitted value from the single fit on all n points
- $h_{ii}$: leverage of point $i$: the $i$-th diagonal of $H$, between 0 and 1

High-leverage points (far from $\bar x$) have $h_{ii}$ close to 1, so their residuals are inflated the most. That is right, because the full fit was pulled toward them. For a straight line, $h_{ii}=\tfrac1n+\tfrac{(x_i-\bar x)^2}{\sum_m (x_m-\bar x)^2}$. Replacing each $h_{ii}$ by the average $\operatorname{tr}(H)/n$ gives **generalized cross-validation** (GCV), $\tfrac1n\sum(y_i-\hat y_i)^2/(1-\operatorname{tr}H/n)^2$. `RidgeCV` uses this trick to try many $\lambda$ for the cost of one fit.

## Worked example: 10 points, 5 folds, a straight line

Fit $\hat y=b_0+b_1x$ to these 10 points with squared-error loss. The folds pair each point with the one five positions later: $F_1=\{1,6\}$, $F_2=\{2,7\}$, …, $F_5=\{5,10\}$. This is one particular shuffle; it keeps every fold spread across the range of $x$.

The data. The least-squares line on all 10 points is ŷ = 0.0267 + 1.9697x, with training MSE 0.2288.

| x | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| y | 2.1 | 3.9 | 6.2 | 7.8 | 9.1 | 12.5 | 13.8 | 15.2 | 18.6 | 19.4 |
| fold | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 |

One row per round: fit on the 8 training points, predict the 2 held-out points, average their squared residuals.

| Fold j | Held out x | Line fitted on the other 8 | Predictions | Residuals | L_j (MSE) |
| --- | --- | --- | --- | --- | --- |
| 1 | 1, 6 | −0.120 + 1.978x | 1.858, 11.750 | 0.242, 0.750 | 0.3105 |
| 2 | 2, 7 | 0.056 + 1.966x | 3.989, 13.820 | −0.089, −0.020 | 0.0042 |
| 3 | 3, 8 | −0.100 + 2.000x | 5.900, 15.900 | 0.300, −0.700 | 0.2900 |
| 4 | 4, 9 | 0.191 + 1.921x | 7.874, 17.478 | −0.074, 1.122 | **0.6324** |
| 5 | 5, 10 | 0.029 + 1.997x | 10.012, 19.996 | −0.912, −0.596 | 0.5938 |

**Mean and standard error**

1. $\mathrm{CV}_5=\tfrac15(0.3105+0.0042+0.2900+0.6324+0.5938)=\mathbf{0.3662}$.
2. Deviations from the mean: $-0.056,\,-0.362,\,-0.076,\,0.266,\,0.228$; $s_L^2=\tfrac14\sum(\cdot)^2=0.0657$, so $s_L=0.2563$.
3. $\widehat{\mathrm{SE}}=0.2563/\sqrt5=\mathbf{0.1146}$. Report $0.366\pm0.115$. The fold losses range from 0.004 to 0.632 just from which two points land together.
4. Compare with the training MSE of 0.229: the honest estimate is 60% higher.

Now **LOOCV** with the shortcut: one fit on all 10 points, then divide each residual by $1-h_{ii}$. Here $\bar x=5.5$ and $\sum(x_m-\bar x)^2=82.5$, so for $x=1$, $h_{11}=0.1+20.25/82.5=0.3455$.

Leave-one-out residuals from one fit (shortcut) versus 10 refits (brute force). They agree to 1e-14.

| x | residual eᵢ | leverage hᵢᵢ | eᵢ / (1 − hᵢᵢ) | brute force |
| --- | --- | --- | --- | --- |
| 1 | 0.1036 | 0.3455 | 0.1583 | 0.1583 |
| 2 | −0.0661 | 0.2485 | −0.0879 | −0.0879 |
| 3 | 0.2642 | 0.1758 | 0.3206 | 0.3206 |
| 4 | −0.1055 | 0.1273 | −0.1208 | −0.1208 |
| 5 | −0.7752 | 0.1030 | −0.8642 | −0.8642 |
| 6 | 0.6552 | 0.1030 | 0.7304 | 0.7304 |
| 7 | −0.0145 | 0.1273 | −0.0167 | −0.0167 |
| 8 | −0.5842 | 0.1758 | −0.7088 | −0.7088 |
| 9 | 0.8461 | 0.2485 | 1.1258 | 1.1258 |
| 10 | −0.3236 | 0.3455 | −0.4944 | −0.4944 |

Squaring and averaging the last column gives $\mathrm{LOOCV}=\mathbf{0.3445}$ (SE $0.133$ over the 10 single-point losses). It is close to the 5-fold value of 0.366 and slightly lower, as expected: each LOO model trains on 9 points instead of 8. The end points $x=1,10$ have the largest leverage, so their residuals are inflated the most (by 53%). The point at $x=9$ has the largest LOO error, 1.126.

## Watch it run

> Figure: 20 points in time order, fitted by a straight line (the true trend is slightly curved). Each row of the grid is one round; orange cells are that round's validation fold. Step through: the line is refitted on the blue points only, its errors on the orange points become one bar, and the dashed mean and ± SE band settle as bars accumulate. Default 5-fold: 2.95 ± 0.67 (×10⁻³) vs training MSE 2.13. Reshuffling moves the 5-fold estimate between 2.59 and 3.15 (seeds 1–5). At k = 20 (LOOCV) you get 2.61 ± 0.57, equal to the one-fit shortcut. Switch to time series: 5 expanding windows give 5.07 ± 3.01, because each fold must extrapolate the curve into the future.

## Splitting schemes

The averaging is always the same; what changes is **how the folds are drawn**. The rule: a validation fold should relate to its training folds the way future data will relate to the data you have.

**Which splitter**

- **KFold (shuffled):** i.i.d. rows. Always `shuffle=True` unless the file order is meaningless *and* random; sorted files (by date, by label) otherwise produce folds that differ systematically.
- **StratifiedKFold:** classification: each fold keeps the overall class ratios. Essential for imbalanced classes. With 5% positives and 100 rows, a plain fold of 20 could contain zero positives and make recall undefined. For regression you can stratify on binned $y$, as the figure does.
- **GroupKFold / StratifiedGroupKFold:** rows come in groups (several images per patient, sessions per user, sentences per document). All rows of a group go to the same fold, so the model is never validated on a patient it trained on. `LeaveOneGroupOut` is the group analogue of LOOCV.
- **TimeSeriesSplit:** ordered data. Fold $j$ trains on everything before a cut and validates on the next block (expanding window; `max_train_size` makes it a rolling window). `gap` drops samples between train and test when features or labels look ahead. The first block is never validated.
- **Repeated (Stratified)KFold:** re-run k-fold with several shuffles and average; reduces the partition noise you see when pressing *reshuffle*.

> **Shuffling a time series:** With shuffled k-fold on time-ordered data, a model trains on Tuesday and Thursday and is validated on Wednesday. Because of autocorrelation that is interpolation, not forecasting, and the score is far too good. The figure shows the reverse: honest forward-chaining gives 5.07 against 2.95 for shuffled folds on the same 20 points. Use `TimeSeriesSplit` (with a `gap` if lagged features or labels span several steps), and never let a fold's training data come after its validation data.

## Nested cross-validation

If you try 50 hyperparameter settings and report the best CV score, that number is **optimistically biased**: you selected the maximum of 50 noisy estimates, so part of its height is luck. This is the winner's curse. The selection is part of the learning procedure, so it must be inside the loop that is being scored.

**Nested CV (outer k₁ × inner k₂)**

1. **Outer loop:** split into $k_1$ folds. For each outer fold $j$, set it aside as the test set.
2. **Inner loop:** on the remaining data only, run an ordinary $k_2$-fold CV for every hyperparameter setting and pick the best (this is just `GridSearchCV`).
3. Refit the chosen setting on all of the outer-training data and score it on outer fold $j$: this gives $L_j^{\text{outer}}$.
4. Average the $k_1$ outer scores. This estimates the performance of *"tune by inner CV, then train"*. The chosen hyperparameters may differ between outer folds, which is fine and shows how stable the choice is.

$$
\mathrm{CV}^{\text{nested}}=\frac{1}{k_1}\sum_{j=1}^{k_1}L\Big(F_j,\ \hat f_{\hat\lambda_j}^{(-j)}\Big),\qquad \hat\lambda_j=\arg\min_{\lambda}\ \mathrm{CV}_{k_2}\big(\lambda;\,D\setminus F_j\big)
$$

- $\hat\lambda_j$: hyperparameters chosen by inner CV using only $D\setminus F_j$
- $\hat f_{\hat\lambda_j}^{(-j)}$: model with those hyperparameters refitted on $D\setminus F_j$
- $k_1,\ k_2$: outer and inner fold counts; cost ≈ k₁ · k₂ · (grid size) fits

Nested CV **evaluates**; it does not produce the final model. To deploy, run the inner search once on all the data and refit. Its `best_score_` is the optimistic number; the nested score is the one to report. If you have plenty of data, a single held-out test set touched once does the same job more cheaply. See *Train/validation/test*.

## In code

```python
import numpy as np

x = np.arange(1, 11, dtype=float)
y = np.array([2.1, 3.9, 6.2, 7.8, 9.1, 12.5, 13.8, 15.2, 18.6, 19.4])
folds = [np.array([j, j + 5]) for j in range(5)]          # {1,6}, {2,7}, ... {5,10}

def fit(xs, ys):                                           # least-squares line
    b1, b0 = np.polyfit(xs, ys, 1)
    return lambda t: b0 + b1 * t

L = []
for val in folds:
    train = np.setdiff1d(np.arange(10), val)
    f = fit(x[train], y[train])                            # fit on train ONLY
    L.append(np.mean((y[val] - f(x[val])) ** 2))           # score on the held-out fold
L = np.array(L)
print(L.round(4))                                          # [0.3105 0.0042 0.29 0.6324 0.5938]
print(L.mean(), L.std(ddof=1) / np.sqrt(len(L)))           # CV_5 = 0.3662, SE = 0.1146

# LOOCV with one fit: e_i / (1 - h_ii)
X = np.c_[np.ones(10), x]
H = X @ np.linalg.solve(X.T @ X, X.T)                      # hat matrix
e = y - H @ y
print(np.mean((e / (1 - np.diag(H))) ** 2))                # 0.3445, same as 10 refits
```

From scratch: the worked example (5-fold, then LOOCV with the leverage shortcut).

```python
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (GroupKFold, KFold, StratifiedKFold,
                                     TimeSeriesSplit, cross_val_score, cross_validate)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))  # scaler refit per fold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy")
print(scores.round(3), f"{scores.mean():.3f} ± {scores.std(ddof=1) / np.sqrt(len(scores)):.3f}")
# [0.956 0.974 0.982 1.    0.982] 0.979 ± 0.007

# The splitter is the only thing that changes between schemes:
KFold(n_splits=5, shuffle=True, random_state=0)   # i.i.d. regression data
StratifiedKFold(n_splits=5)                       # classification: keep class ratios per fold
GroupKFold(n_splits=5)                            # pass groups=...: a group never spans train and val
TimeSeriesSplit(n_splits=5, gap=0)                # train on the past, validate on the next block

groups = np.arange(len(y)) // 4                   # e.g. 4 scans per patient
res = cross_validate(model, X, y, groups=groups, cv=GroupKFold(n_splits=5),
                     scoring=["accuracy", "roc_auc"], return_train_score=True)
print(res["train_accuracy"].mean().round(3), res["test_accuracy"].mean().round(3))  # 0.989 0.975
```

scikit-learn: put preprocessing inside the pipeline, and choose the splitter to match the data.

```python
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), SVC())
grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.01, 0.001]}

inner = StratifiedKFold(5, shuffle=True, random_state=1)    # picks C, gamma
outer = StratifiedKFold(5, shuffle=True, random_state=2)    # scores the whole procedure
search = GridSearchCV(pipe, grid, cv=inner, scoring="accuracy")

nested = cross_val_score(search, X, y, cv=outer)            # search re-run inside every outer fold
print(f"nested CV accuracy {nested.mean():.3f} ± {nested.std(ddof=1) / np.sqrt(5):.3f}")  # 0.975 ± 0.012

search.fit(X, y)                                            # final model: search on all data
print(search.best_params_, f"best inner score {search.best_score_:.3f} (optimistic)")      # C=10, gamma=0.01, 0.982
```

Nested CV: GridSearchCV is the inner loop, cross_val_score the outer one.

> **Preprocessing outside the pipeline leaks:** `StandardScaler().fit(X)` before `cross_val_score` lets every validation fold influence the means and variances used to train on it. With mild preprocessing the leak is small, but for **feature selection** it is catastrophic: pick the 20 genes most correlated with the label on all the data, then cross-validate, and pure noise will score far above chance. Imputation, scaling, encoding, resampling (SMOTE), PCA and target encoding all belong *inside* a `Pipeline`, so they are refitted on each training fold.

**Strengths**

- Data is limited and a single validation split would be too small or too noisy.
- You need an error bar, or a fair comparison of models on identical folds (use the same `cv` object, then compare per-fold differences).
- Choosing hyperparameters (`GridSearchCV`, `RandomizedSearchCV`), and nested CV to report the tuned procedure honestly.
- Linear smoothers: LOOCV / GCV is essentially free.

**Limitations**

- Training is expensive (large deep nets): k fits cost k times as much. Use one held-out validation set instead.
- The data is huge: a single 1% validation split already has a tiny standard error.
- The folds cannot mimic deployment (distribution shift, new users, the future) unless you pick the matching group or time splitter.
- You need the error of *the* deployed model: CV scores the procedure, averaged over training sets.

- **Model selection:** Choose the regularization strength, tree depth, k in kNN, or number of components by minimizing CV error (often with the one-SE rule).
- **Small-sample science:** Medical, genomics and psychology studies with a few hundred subjects, using grouped and nested CV so subjects never leak.
- **Forecasting backtests:** Rolling-origin evaluation of demand or price forecasts is TimeSeriesSplit under another name.
- **Stacking:** Out-of-fold predictions from `cross_val_predict` are the honest inputs for training a meta-model.

> **Duplicates and near-duplicates:** If the same item appears several times (augmented copies, repeated measurements, a customer's many rows), plain k-fold puts copies on both sides of the split and rewards memorization. Find the unit of independence and split by it with `GroupKFold`.

> **Tuning to the CV score:** Running many experiments against the same folds slowly overfits them, exactly like overfitting a validation set. Keep a final test set you look at once, or use nested CV.

**Question:** Five fold losses are $0.3105, 0.0042, 0.2900, 0.6324, 0.5938$. What are $\mathrm{CV}_5$ and its standard error?

- A. 0.366 and 0.256
- B. 0.366 and 0.115
- C. 0.290 and 0.115
- D. 0.366 and 0.051

**Answer:** B. 0.366 and 0.115 — The mean is 0.366. The sample standard deviation of the five losses is 0.256, and dividing by $\sqrt5$ gives SE = 0.115. Because the folds share training data, the true uncertainty is somewhat larger than this.

**Question:** Compared with 10-fold CV, leave-one-out CV typically has…

- A. More bias and less variance
- B. Less bias, but often more variance, because its n training sets are almost identical
- C. Less bias and less variance, always
- D. The same bias and variance but higher cost

**Answer:** B. Less bias, but often more variance, because its n training sets are almost identical — Each LOO model trains on n − 1 points, so the pessimistic bias nearly vanishes. But the n models are highly correlated, so averaging them reduces variance much less than n independent estimates would.

**Question:** An OLS fit on all data has residual $e_i=0.30$ at a point with leverage $h_{ii}=0.25$. Its leave-one-out residual is…

- A. 0.225
- B. 0.30
- C. 0.40
- D. 1.20

**Answer:** C. 0.40 — $e_i/(1-h_{ii}) = 0.30/0.75 = 0.40$. The point pulled the full fit toward itself; removing it shows the larger, honest error, with no refit needed.

**Question:** You standardize all features, select the 50 most label-correlated ones on the full dataset, then run 10-fold CV. The CV accuracy is…

- A. Unbiased, since the model never saw the validation labels
- B. Optimistically biased: the selection used the validation folds' labels
- C. Pessimistically biased, because of fewer features
- D. Unaffected as long as folds are stratified

**Answer:** B. Optimistically biased: the selection used the validation folds' labels — Feature selection looked at every label, including those in each validation fold, so information leaked across the split. Put the scaler and the selector inside a Pipeline so they are refitted on each training fold only.

**Key points**

- k-fold CV rotates the held-out fold so each sample is validated exactly once: $\mathrm{CV}_k=\tfrac1k\sum_j L_j$, refit on all data afterwards.
- Report mean ± $s_L/\sqrt k$, but remember fold losses are correlated, so this SE is an optimistic, approximate error bar.
- Small k is pessimistically biased; LOOCV is nearly unbiased but can have high variance. 5–10 folds is the usual sweet spot; repeat with new shuffles for stability.
- For linear smoothers LOOCV needs one fit: residual $/(1-h_{ii})$; GCV replaces $h_{ii}$ by $\operatorname{tr}H/n$.
- Match the splitter to the data: stratified for classes, group for repeated units, TimeSeriesSplit (never shuffled) for ordered data.
- Put all preprocessing inside the pipeline, and use nested CV (or an untouched test set) to report a tuned model honestly.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
