# Feature scaling & encoding

> Distance- and gradient-based models need features on comparable scales, and categories must become numbers, e.g. one-hot vectors.

Source: https://www.mlfoundry.in/learn/feature-scaling-encoding  
Section: ML Algorithms › Evaluation & Generalization  
Level: core · 20 min read · updated 2026-10-08

**Feature scaling:** re-expresses each numeric column in comparable units, typically by subtracting a centre and dividing by a spread, so that no feature dominates a distance, a gradient or a penalty just because of the units it was recorded in.

**Categorical encoding:** turns a non-numeric column (city, colour, product ID) into one or more numeric columns that a model can do arithmetic on, without inventing an order or a distance the data does not have.

- **Stage:** preprocessing: fitted on the training data, then applied unchanged to validation, test and production data
- **Learns:** per column: a centre and a scale (mean/σ, min/range, median/IQR), or a category → code mapping
- **Who needs scaling:** kNN, k-means, SVM (RBF), PCA, linear/logistic regression trained by gradient descent, ridge/lasso, neural networks
- **Who doesn't:** decision trees, random forests, gradient-boosted trees: a split threshold is unaffected by any monotonic transform
- **Main encoders:** one-hot, ordinal, target (mean) encoding with smoothing, frequency, hashing
- **Main risk:** leakage: fitting the scaler or target encoder on rows you later evaluate on

## The problem

A house has an area of 80 m² and 1 room. Another has 75 m² and 3 rooms. Which one is "closer" to a third house? A Euclidean distance adds $(\Delta\text{area})^2 + (\Delta\text{rooms})^2$, so a 5 m² difference in area counts as much as a 5-room difference. The answer depends on whether you measured area in m², ft² or km², which is absurd. **Units are an accident of bookkeeping, and models that combine features arithmetically inherit that accident.**

**Why each model family cares (or not)**

- **Distance-based:** kNN, k-means, RBF-SVM and DBSCAN compute $\lVert x-x'\rVert$. The feature with the largest numeric range decides who is a neighbour; the rest are noise.
- **Variance-based:** PCA finds directions of maximal variance. Area in m² has variance ≈ 4000; rooms ≈ 1.7. The first component is "area", whatever the correlations say.
- **Gradient-based:** Linear and logistic regression, neural nets. Unequal scales stretch the loss surface into a narrow valley; gradient descent zig-zags across it and crawls along it.
- **Penalized:** Ridge and lasso penalize $\lVert w\rVert$. A feature in small units needs a large weight and is penalized more, for no real reason.
- **Trees:** A split asks *is $x_j\le t$?*. Rescaling $x_j$ just moves $t$: the same rows go left. Trees are **invariant** to monotonic transforms, so scaling changes nothing.

Encoding has the opposite problem. "Paris", "Lyon" and "Nice" are not numbers, and most models (all of scikit-learn's, except a few) only accept numbers. Writing Lyon = 0, Nice = 1, Paris = 2 makes the data numeric, but a linear model then believes Paris is "two Nices" and that Nice lies between Lyon and Paris. A good encoding gives the model numbers **without inventing an order or a distance** the categories don't have.

## The scalers

All the common scalers are affine: $x' = (x - c)/s$ with a centre $c$ and scale $s$ computed **per column on the training set**. They differ only in which statistics they use, and that decides how they react to outliers.

$$
z=\frac{x-\mu}{\sigma},\qquad \mu=\frac1n\sum_{i}x_i,\quad \sigma=\sqrt{\frac1n\sum_i (x_i-\mu)^2}
$$

- $z$: standardized value (a z-score): how many standard deviations x is from the mean
- $\mu$: column mean on the training set
- $\sigma$: column standard deviation (scikit-learn uses the population form, dividing by n)

**Standardization** (StandardScaler) gives every column mean 0 and variance 1. It is the default for linear models, SVMs, PCA and neural nets. It does *not* make data Gaussian and has no fixed range; since $\mu$ and $\sigma$ are both pulled by outliers, one extreme value shrinks everyone else's z-scores.

$$
x'=\frac{x-\min(x)}{\max(x)-\min(x)}\in[0,1]
$$

- $\min(x),\ \max(x)$: smallest and largest training values of the column
- $x'$: min-max scaled value; 0 at the training minimum, 1 at the maximum

**Min-max scaling** (MinMaxScaler) maps the training range onto $[0,1]$. It is handy when a model or a pixel-like input expects a bounded range, but it is the **most fragile** scaler: the range is set by the two most extreme points, so a single outlier compresses every other value into a sliver near 0. New data outside the training range lands outside $[0,1]$.

$$
x'=\frac{x-\operatorname{median}(x)}{Q_3(x)-Q_1(x)}
$$

- $\operatorname{median}(x)$: 50th percentile of the training column
- $Q_1,\ Q_3$: 25th and 75th percentiles; their difference is the interquartile range (IQR)

**Robust scaling** (RobustScaler) uses statistics that ignore the tails. Moving the largest value from 240 to 600 m² does not change the median or the quartiles at all, so the inliers keep their scaled values. The outlier itself is not tamed: it simply ends up far away, where it belongs.

$$
x'=\frac{x}{\max_i |x_i|}\in[-1,1]
$$

- $\max_i|x_i|$: largest absolute training value; no centring

**Max-abs scaling** (MaxAbsScaler) only divides. Because it never subtracts a centre, zeros stay zeros, so it is the scaler for **sparse** matrices (bag-of-words counts, one-hot blocks) where centring would turn millions of zeros into non-zeros. It is as outlier-sensitive as min-max.

## Fixing the shape, not just the scale

Affine scalers change units, never shape: a right-skewed column (incomes, prices, counts) stays skewed, with a long tail that still dominates distances and squared errors. A **monotonic non-linear transform** compresses the tail first. Trees don't care about these either; linear models and kNN often do.

$$
x^{(\lambda)}=\begin{cases}\dfrac{x^{\lambda}-1}{\lambda}, & \lambda\neq 0\\[4pt] \ln x, & \lambda=0\end{cases}\qquad (x>0)
$$

- $\lambda$: power parameter, fitted by maximum likelihood so the result is as Gaussian as possible
- $\ln x$: the $\lambda\to0$ limit: the plain log transform

**Log** (or $\log(1+x)$ for counts with zeros) is the everyday choice for multiplicative quantities: it turns ratios into differences. **Box-Cox** (above) learns the best power, but needs strictly positive data. **Yeo-Johnson** extends it to zero and negative values by applying a Box-Cox-like power to $x+1$ for $x\ge0$ and to $1-x$ (with power $2-\lambda$) for $x<0$. scikit-learn's PowerTransformer fits either and standardizes the result; QuantileTransformer goes further and maps each column through its empirical CDF onto a uniform or normal distribution.

## Why gradient descent needs scaled features

For least-squares regression $L(w)=\frac{1}{2n}\lVert Xw-y\rVert^2$, the loss is a quadratic bowl whose curvature is the Hessian:

$$
\nabla^2 L = H=\tfrac1n X^\top X,\qquad \kappa(H)=\frac{\lambda_{\max}(H)}{\lambda_{\min}(H)}
$$

- $H$: Hessian: curvature of the loss; constant for a quadratic
- $\lambda_{\max},\ \lambda_{\min}$: largest and smallest eigenvalues: curvature along the steepest and flattest directions
- $\kappa$: condition number: how elongated the elliptical loss contours are (axis ratio √κ)

If one column is measured in large units its variance is huge, so $H$ has one enormous eigenvalue and the contours become a long, thin valley. Gradient descent must keep the step below $2/\lambda_{\max}$ or it diverges along the steep direction, and with such a small step progress along the flat direction is glacial. Even with the best fixed step, the error shrinks per iteration by:

$$
\lVert w_{t}-w^\star\rVert \le \left(\frac{\kappa-1}{\kappa+1}\right)^{t}\lVert w_0-w^\star\rVert \quad\Rightarrow\quad t\approx\frac{\kappa}{2}\ln\frac{1}{\varepsilon}\ \text{iterations}
$$

- $w^\star$: the least-squares optimum
- $t$: iteration count
- $\varepsilon$: target relative error

Iterations grow **linearly in κ**. Standardizing divides each column by its σ, which rescales $H$ to the **correlation matrix** of the features: all diagonal entries become 1 and the only elongation left comes from genuine correlation between features. Centring also matters when there is an intercept: an uncentred column of values near 100 is nearly parallel to the all-ones column, which is another source of ill-conditioning. (Normal equations and second-order methods are much less affected, but finite-precision arithmetic still prefers well-conditioned matrices.)

> **On the six houses below:** With columns $[1,\ \text{area},\ \text{rooms}]$, $\kappa(X^\top X)=131{,}766$. Standardizing both features gives $\kappa=15.7$ (what's left is the 0.88 correlation between area and rooms). Gradient descent with step $1/\lambda_{\max}$ needs **668,829 iterations** on the raw columns to get the loss within $10^{-6}$ of optimal, and **59** on the standardized ones. Same data, same model, same predictions at convergence.

## Worked example: six houses

Six houses with two numeric features and one categorical feature, plus a price we will use for target encoding. The query house **Q** has 80 m² and 1 room; we want its nearest neighbour.

The training data. F is a large house that acts as an outlier in area.

| House | Area (m²) | Rooms | City | Price (k€) |
| --- | --- | --- | --- | --- |
| A | 45 | 1 | Paris | 300 |
| B | 60 | 2 | Lyon | 200 |
| C | 75 | 3 | Paris | 390 |
| D | 90 | 3 | Lyon | 240 |
| E | 110 | 4 | Paris | 360 |
| F | 240 | 5 | Nice | 520 |
| **Q** (query) | 80 | 1 | ? | ? |

**Fitting each scaler by hand (training rows A–F only)**

1. **Standard.** Area: $\mu=620/6=103.33$; squared deviations sum to $24{,}983.3$, so $\sigma=\sqrt{24983.3/6}=64.53$. Rooms: $\mu=3$, $\sigma=\sqrt{10/6}=1.291$. House A: $z=\big((45-103.33)/64.53,\ (1-3)/1.291\big)=(-0.904,\ -1.549)$.
2. **Min-max.** Area: min 45, max 240, range 195. Rooms: min 1, range 4. A → $(0, 0)$, E → $(65/195,\ 3/4)=(0.333,\ 0.75)$, F → $(1, 1)$.
3. **Robust.** Sorted areas 45, 60, 75, 90, 110, 240: median $82.5$, $Q_1=63.75$, $Q_3=105$ (linear interpolation), IQR $41.25$. Rooms: median 3, $Q_1=2.25$, $Q_3=3.75$, IQR $1.5$. A → $(-0.909,\ -1.333)$; F → $(3.818,\ 1.333)$.
4. **Max-abs.** Divide by 240 and 5: A → $(0.188,\ 0.2)$, F → $(1, 1)$.
5. **Transform the query with the same numbers**: Q → standard $(-0.362,\ -1.549)$, min-max $(0.179,\ 0)$, robust $(-0.061,\ -1.333)$, max-abs $(0.333,\ 0.2)$. Q is never used to fit anything.

Scaled coordinates (area′, rooms′) of every house.

| House | Standard | Min-max | Robust | Max-abs |
| --- | --- | --- | --- | --- |
| A | (−0.904, −1.549) | (0.000, 0.00) | (−0.909, −1.333) | (0.188, 0.2) |
| B | (−0.672, −0.775) | (0.077, 0.25) | (−0.545, −0.667) | (0.250, 0.4) |
| C | (−0.439, 0.000) | (0.154, 0.50) | (−0.182, 0.000) | (0.313, 0.6) |
| D | (−0.207, 0.000) | (0.231, 0.50) | (0.182, 0.000) | (0.375, 0.6) |
| E | (0.103, 0.775) | (0.333, 0.75) | (0.667, 0.667) | (0.458, 0.8) |
| F | (2.118, 1.549) | (1.000, 1.00) | (3.818, 1.333) | (1.000, 1.0) |
| Q | (−0.362, −1.549) | (0.179, 0.00) | (−0.061, −1.333) | (0.333, 0.2) |

Euclidean distance from Q to each house. Bold = nearest neighbour. Without scaling, rooms are invisible.

| House | Raw | Standard | Min-max | Robust | Max-abs |
| --- | --- | --- | --- | --- | --- |
| A | 35.000 | **0.542** | **0.179** | 0.848 | **0.146** |
| B | 20.025 | 0.834 | 0.270 | **0.824** | 0.217 |
| C | **5.385** | 1.551 | 0.501 | 1.339 | 0.401 |
| D | 10.198 | 1.557 | 0.503 | 1.355 | 0.402 |
| E | 30.150 | 2.370 | 0.766 | 2.128 | 0.613 |
| F | 160.050 | 3.968 | 1.294 | 4.707 | 1.041 |

On raw features Q's nearest neighbour is **C** (75 m², 3 rooms): the 5 m² gap in area is all that counts, and the 2-room gap adds only $\sqrt{25+4}-5=0.39$ to the distance. After standardization a 2-room difference is worth as much as a 100 m² difference ($2/1.291=1.549$ standard deviations either way), and the nearest neighbour becomes **A** (45 m², 1 room, the only other one-room house). Min-max and max-abs agree. Robust scaling divides area by its IQR (41.25) rather than σ (64.53), so area counts relatively more and **B** (20 m² and 1 room away) edges out A (35 m² away, same rooms): 0.824 vs 0.848. A 1-NN price prediction for Q changes from 390 k€ to 300 k€ (or 200 k€). **The choice of scaler is a modelling decision**: it decides how many m² one room is worth.

Now make F a real outlier, 600 m². Min-max squeezes houses A–E into area′ $\in[0,\ 0.117]$ (down from $[0,\ 0.333]$), so area barely matters any more. Standardization shrinks their spread from 1.007 to 0.331 because $\sigma$ grows. Robust scaling leaves A–E exactly where they were (spread 1.576) and sends F to 12.5: the median and quartiles never looked at F.

> Figure: Three views. Scalers: the six houses with each scaler applied, drawn with equal axis scale so the dashed circle really is the set of points at Q's nearest-neighbour distance. Drag F along the raw-area strip and watch min-max crush A–E while robust scaling leaves them alone. Gradient descent: one quadratic loss on raw features (spreads 4 and 1, correlation 0.25, κ = 17.2) and the same loss after standardizing (κ = 1.67), from the same starting model; the raw path zig-zags across the valley and needs 41 steps where the standardized one needs 9 (step 1.7/λmax). Encoding: the city column under each encoder, with a slider for the target-encoding smoothing m.

## Encoding categories

$$
\text{onehot}(c)=\big(\mathbb 1[c=v_1],\ \mathbb 1[c=v_2],\ \dots,\ \mathbb 1[c=v_K]\big)\in\{0,1\}^K
$$

- $v_1,\dots,v_K$: the K categories seen in training
- $\mathbb 1[\cdot]$: indicator: 1 if true, else 0; exactly one entry is 1

**One-hot encoding** gives each category its own 0/1 column. All categories are equidistant (any two differ in exactly two coordinates, distance $\sqrt2$), so no order is invented. A linear model learns one coefficient per category: a per-city offset. Cost: $K$ columns, which is fine for 3 cities and painful for 30,000 ZIP codes.

> **The dummy-variable trap:** The $K$ one-hot columns always sum to 1, which is exactly the intercept column. With an intercept, $X$ has linearly dependent columns, $X^\top X$ is singular and ordinary least squares has no unique solution (any constant can move between the intercept and the city weights). Fix: **drop one category** as the reference (drop="first"), so the remaining coefficients read as "difference from Lyon". Regularized models (ridge, lasso, neural nets) and trees don't need this, and dropping a column makes the encoding asymmetric for them, so keep all $K$ there.

**Ordinal encoding** maps categories to integers $0,1,\dots,K-1$. It is right when the categories *are* ordered (small < medium < large; bad < ok < good) and you choose the order. For unordered categories it is fine for trees, which can isolate any code with two splits, and harmful for linear models and kNN, which read the codes as quantities.

$$
\operatorname{TE}(c)=\frac{n_c\,\bar y_c+m\,\bar y}{n_c+m}=\frac{\sum_{i:\,x_i=c}y_i+m\,\bar y}{n_c+m}
$$

- $n_c$: number of training rows in category c
- $\bar y_c$: mean target among those rows
- $\bar y$: global mean target: the prior
- $m$: smoothing strength: the prior counts as m pseudo-rows

**Target (mean) encoding** replaces each category by the average target of its rows: one dense column, however many categories. The **m-estimate** above shrinks small categories toward the global mean: with $n_c\gg m$ the category speaks for itself; with $n_c\ll m$ it is mostly the prior. It is a Bayesian posterior mean under a prior worth $m$ observations. For classification, $y$ is the 0/1 label and $\operatorname{TE}$ a smoothed class rate.

> **Target encoding leaks the label:** If a row's own target is part of the mean that encodes it, the feature contains the answer. A category with one row gets *exactly* its own label as its code. On pure-noise data with 500 random ZIP codes, naively target-encoding on all rows and then cross-validating a ridge model gives $R^2=0.45$ (the true value is 0); scikit-learn's TargetEncoder gives $-0.01$. The fix is **cross-fitting**: split the training data into folds and encode each fold with statistics from the *other* folds. TargetEncoder.fit_transform does this automatically (5 folds by default); its plain transform, used on new data, uses the full-training statistics.

**Frequency (count) encoding** replaces a category by its share (or count) of training rows. It needs no target, so it cannot leak, and it captures "popular vs rare", but two categories with the same count collide. **Hashing** maps each category through a hash function into one of $D$ buckets, with no stored vocabulary at all:

$$
\phi(c)_j=\sum_{c\,:\,h(c)=j}s(c),\qquad h:\text{strings}\to\{0,\dots,D-1\},\ \ s(c)\in\{-1,+1\}
$$

- $D$: number of hash buckets (output columns), chosen by you, e.g. 2¹⁸
- $h$: hash function deciding the bucket
- $s$: second hash giving a random sign, so collisions cancel on average instead of adding up

Hashing (FeatureHasher) handles unseen and unbounded vocabularies (user IDs, URLs, word n-grams) in fixed memory, at the price of collisions and unreadable columns. Neural networks usually go one step further and *learn* an **embedding** vector per category.

## Worked example: encoding the city

City under four encoders. One-hot uses the alphabetical order Lyon, Nice, Paris; drop-first removes Lyon, the reference.

| House | City | One-hot (L, N, P) | Drop-first (N, P) | Ordinal | Frequency |
| --- | --- | --- | --- | --- | --- |
| A | Paris | (0, 0, 1) | (0, 1) | 2 | 0.500 |
| B | Lyon | (1, 0, 0) | (0, 0) | 0 | 0.333 |
| C | Paris | (0, 0, 1) | (0, 1) | 2 | 0.500 |
| D | Lyon | (1, 0, 0) | (0, 0) | 0 | 0.333 |
| E | Paris | (0, 0, 1) | (0, 1) | 2 | 0.500 |
| F | Nice | (0, 1, 0) | (1, 0) | 1 | 0.167 |

**Target encoding with smoothing m = 2**

1. Global mean price: $\bar y=(300+200+390+240+360+520)/6=2010/6=335$.
2. **Paris** (A, C, E): $n=3$, sum $1050$, raw mean $350$. $\operatorname{TE}=(1050+2\cdot335)/(3+2)=1720/5=344.0$.
3. **Lyon** (B, D): $n=2$, sum $440$, raw mean $220$. $\operatorname{TE}=(440+670)/4=277.5$.
4. **Nice** (F only): $n=1$, raw mean $520$. $\operatorname{TE}=(520+670)/3=396.7$. Without smoothing ($m=0$) Nice's code would be 520, which is F's own price: a perfect leak.
5. Smoothing moves each code toward 335 in proportion to $m/(n_c+m)$: Paris by 40%, Lyon by 50%, Nice by 67%. The rarest category is trusted least.
6. Cross-fitted instead (3 unshuffled folds {A,B}, {C,D}, {E,F}): F is encoded from A–D only. Nice never appears there, so F gets the fold's prior, $282.5$, and its own price cannot leak.

## In code

```python
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, OneHotEncoder, TargetEncoder

houses = pd.DataFrame({
    "area":  [45, 60, 75, 90, 110, 240],
    "rooms": [1, 2, 3, 3, 4, 5],
    "city":  ["Paris", "Lyon", "Paris", "Lyon", "Paris", "Nice"],
    "price": [300, 200, 390, 240, 360, 520],
}, index=list("ABCDEF"))
X, q = houses[["area", "rooms"]], pd.DataFrame({"area": [80], "rooms": [1]})

for scaler in [None, StandardScaler(), MinMaxScaler(), RobustScaler()]:
    Z, Zq = (X.to_numpy(), q.to_numpy()) if scaler is None else (scaler.fit_transform(X), scaler.transform(q))
    d = np.linalg.norm(Z - Zq, axis=1)
    print(f"{type(scaler).__name__:15s} nearest = {houses.index[d.argmin()]}  d = {d.round(3)}")
# NoneType        nearest = C  d = [ 35.     20.025   5.385  10.198  30.15  160.05 ]
# StandardScaler  nearest = A  d = [0.542 0.834 1.551 1.557 2.37  3.968]
# MinMaxScaler    nearest = A  d = [0.179 0.27  0.501 0.503 0.766 1.294]
# RobustScaler    nearest = B  d = [0.848 0.824 1.339 1.355 2.128 4.707]

print(OneHotEncoder(drop="first").fit(houses[["city"]]).get_feature_names_out())  # ['city_Nice' 'city_Paris']
te = TargetEncoder(smooth=2.0, target_type="continuous").fit(houses[["city"]], houses.price)
print(dict(zip(te.categories_[0], te.encodings_[0].round(2))))  # Lyon 277.5, Nice 396.67, Paris 344.0
```

Reproducing the worked example with scikit-learn. The printed numbers match the tables above.

```python
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
n = 600
df = pd.DataFrame({
    "area":  rng.lognormal(4.4, 0.4, n).round(),        # m², roughly 40-250
    "rooms": rng.integers(1, 7, n),
    "city":  rng.choice(["Paris", "Lyon", "Nice", "Lille"], n, p=[0.4, 0.3, 0.2, 0.1]),
})
df["price"] = (2.5 * df.area + 15 * df.rooms
               + df.city.map({"Paris": 150, "Lyon": 40, "Nice": 90, "Lille": 0})
               + rng.normal(0, 25, n))
X, y = df.drop(columns="price"), df.price

def knn(numeric):
    pre = ColumnTransformer([
        ("num", numeric, ["area", "rooms"]),
        ("cat", OneHotEncoder(handle_unknown="ignore"), ["city"]),
    ])
    return Pipeline([("pre", pre), ("knn", KNeighborsRegressor(n_neighbors=10))])

# cross_val_score re-fits the scaler and encoder on each training fold: no leakage.
print("raw    R2:", cross_val_score(knn("passthrough"), X, y, cv=5).mean().round(3))     # 0.709
print("scaled R2:", cross_val_score(knn(StandardScaler()), X, y, cv=5).mean().round(3))  # 0.903

model = knn(StandardScaler()).fit(X, y)
new = pd.DataFrame({"area": [80], "rooms": [3], "city": ["Marseille"]})  # a city never seen in training
print(model.predict(new))  # works: an unknown city becomes an all-zero one-hot block
```

The production pattern: ColumnTransformer inside a Pipeline, so every preprocessing statistic is fitted on training folds only.

```python
import numpy as np, pandas as pd
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(1)
zips = rng.integers(0, 500, 1000).astype(str)       # 500 categories, ~2 rows each
y = rng.normal(0, 1, 1000)                          # no relation to zip at all

# Leaky: category means computed on ALL rows, before cross-validation
leaky = pd.DataFrame({"zip_te": pd.Series(zips).map(pd.Series(y).groupby(zips).mean())})
print(cross_val_score(Ridge(), leaky, y, cv=5).mean().round(3))                     # 0.45 (fake)

# Correct: TargetEncoder inside the pipeline, cross-fitted on each training fold
good = make_pipeline(TargetEncoder(target_type="continuous"), Ridge())
print(cross_val_score(good, pd.DataFrame({"zip": zips}), y, cv=5).mean().round(3))  # -0.008
```

Why naive target encoding is dangerous: on a target that is pure noise it fakes a strong signal.

## Which scaler, which encoder?

Rules of thumb. When in doubt, cross-validate the choice: it is a hyperparameter like any other.

| Situation | Use | Why |
| --- | --- | --- |
| Linear / logistic models, SVM, PCA, neural nets | StandardScaler | zero mean, unit variance: well-conditioned gradients and fair penalties |
| Heavy outliers you want to keep | RobustScaler | median and IQR ignore the tails, so inliers keep their spread |
| Bounded inputs expected (images, some activations) | MinMaxScaler | exact [0, 1] range; only if there are no wild outliers |
| Sparse data (text counts, one-hot blocks) | MaxAbsScaler (or none) | no centring keeps zeros zero |
| Right-skewed positive data (income, price, counts) | log / log1p, Box-Cox, then standardize | compress the tail before choosing a scale |
| Trees, random forests, gradient boosting | no scaling | splits are invariant to monotonic transforms |
| Few unordered categories (< ~15) | OneHotEncoder | no invented order; drop='first' only for unregularized linear models |
| Genuinely ordered categories | OrdinalEncoder with an explicit order | keeps the order in one column |
| High cardinality (ZIP, product ID) with a target | TargetEncoder (cross-fitted, smoothed) | one dense column; smoothing tames rare categories |
| Huge or open vocabulary, streaming | FeatureHasher | fixed memory, no vocabulary, handles unseen values |
| Tree models with categories | native categorical support (LightGBM, CatBoost, HistGradientBoosting) or ordinal codes | trees split on codes directly |

**Strengths**

- Scale when the model uses distances, dot products, variances, gradients or norm penalties: kNN, k-means, SVM, PCA, linear models, neural nets.
- Standardization makes gradient descent converge in tens of steps instead of hundreds of thousands, and makes coefficient sizes comparable.
- One-hot is the safe default for low-cardinality nominal features: no assumptions, interpretable coefficients.
- Target encoding compresses thousands of categories into one informative column, which is often the best feature in tabular data.

**Limitations**

- Skip scaling for tree ensembles: it cannot help, and it makes the stored thresholds harder to read.
- Scaling makes features comparable in units, not in importance: it weights every feature equally, which may drown a crucial feature among noisy ones.
- One-hot explodes with high cardinality and creates rare, noisy columns; hashing and target encoding trade interpretability for size.
- Target encoding needs cross-fitting and smoothing; done naively it leaks the label and produces optimistic validation scores.

> **Fitting the scaler on all the data:** Calling fit_transform on the full dataset before splitting lets test-set means, ranges and category statistics shape the training features. For scalers the leak is usually small, for target encoding it can be enormous, and in time series it lets the future leak into the past. Always fit preprocessing on the training split only, ideally by putting it inside a Pipeline so cross-validation does this for you.

> **Unseen and rare categories:** Production data contains categories the training set never saw (a new city, a new product). OneHotEncoder raises an error by default; set handle_unknown='ignore' (all-zero block) or 'infrequent_if_exist' together with min_frequency to pool rare categories into one 'other' column. TargetEncoder maps unseen categories to the global mean.

> **Scaling the encoded columns:** Encoders produce columns on their own scales: one-hot columns are 0/1, target codes are in units of the target (here k€). For distance-based models put a StandardScaler after a TargetEncoder, or one code column can outweigh all the others.

**Question:** Which of these models gives exactly the same predictions whether or not you standardize the numeric features?

- A. k-nearest neighbours
- B. Ridge regression
- C. Random forest
- D. PCA followed by logistic regression

**Answer:** C. Random forest — A tree split x ≤ t sends the same rows left after any monotonic rescaling, because the threshold moves with the data. kNN distances, ridge penalties and PCA variances all depend on the units.

**Question:** House F's area grows from 240 to 600 m² while A–E stay the same. Which scaler leaves the scaled areas of A–E unchanged?

- A. MinMaxScaler
- B. StandardScaler
- C. RobustScaler
- D. MaxAbsScaler

**Answer:** C. RobustScaler — The median (82.5) and the quartiles (63.75, 105) do not depend on how large the maximum is. Min-max and max-abs divide by a range set by F; standardization's μ and σ both move with F.

**Question:** Nice has one house, priced 520 k€; the global mean price is 335 k€. What is Nice's target encoding with smoothing $m=2$?

- A. 520.0
- B. 427.5
- C. 396.7
- D. 335.0

**Answer:** C. 396.7 — $\operatorname{TE}=(1\cdot520+2\cdot335)/(1+2)=1190/3=396.7$. The single observation counts once, the prior twice, so the code sits two thirds of the way back toward the global mean.

**Question:** You one-hot encode a 4-category feature into 4 columns and fit ordinary least squares with an intercept. What goes wrong?

- A. Nothing; OLS handles it
- B. The four columns sum to the intercept column, so XᵀX is singular and the coefficients are not unique
- C. The model can no longer represent the category with the largest mean
- D. Gradient descent diverges because the columns are not standardized

**Answer:** B. The four columns sum to the intercept column, so XᵀX is singular and the coefficients are not unique — This is the dummy-variable trap. Drop one category (it becomes the reference), drop the intercept, or add regularization.

**Key points**

- Models that use distances, variances, dot products, gradients or norm penalties are sensitive to units; trees are not.
- Standardize $(x-\mu)/\sigma$ by default; min-max for bounded inputs without outliers; robust (median/IQR) when outliers would squash everyone else; max-abs for sparse data; log/Box-Cox/Yeo-Johnson to fix skew.
- Unscaled features make $X^\top X$ ill-conditioned: elongated contours, zig-zagging gradient descent, iterations ∝ κ. Six houses: κ 131,766 → 15.7 and 668,829 → 59 iterations.
- Scaling changes who your nearest neighbours are, so the scaler is a modelling choice: in the example Q's neighbour moved from C to A (or B).
- One-hot for few nominal categories (drop one only for unregularized linear models); ordinal only for truly ordered ones; target encoding with smoothing and cross-fitting for high cardinality; hashing for open vocabularies.
- Fit every scaler and encoder on training data only, inside a Pipeline; plan for unseen categories.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
