Feature scaling & encoding

Distance- and gradient-based models need features on comparable scales, and categories must become numbers, e.g. one-hot vectors.

20 min read2023 wordscore0 viewsupdated 2026-10-08

Definition

Feature scaling re-expresses each numeric column in comparable units, typically by subtracting a centre and dividing by a spread, so that no feature dominates a distance, a gradient or a penalty just because of the units it was recorded in.

Definition

Categorical encoding turns a non-numeric column (city, colour, product ID) into one or more numeric columns that a model can do arithmetic on, without inventing an order or a distance the data does not have.

At a glance

Stage
preprocessing: fitted on the training data, then applied unchanged to validation, test and production data
Learns
per column: a centre and a scale (mean/σ, min/range, median/IQR), or a category → code mapping
Who needs scaling
kNN, k-means, SVM (RBF), PCA, linear/logistic regression trained by gradient descent, ridge/lasso, neural networks
Who doesn't
decision trees, random forests, gradient-boosted trees: a split threshold is unaffected by any monotonic transform
Main encoders
one-hot, ordinal, target (mean) encoding with smoothing, frequency, hashing
Main risk
leakage: fitting the scaler or target encoder on rows you later evaluate on

The problem

A house has an area of 80 m² and 1 room. Another has 75 m² and 3 rooms. Which one is "closer" to a third house? A Euclidean distance adds (Δarea)2+(Δrooms)2(\Delta\text{area})^2 + (\Delta\text{rooms})^2, so a 5 m² difference in area counts as much as a 5-room difference. The answer depends on whether you measured area in m², ft² or km², which is absurd. Units are an accident of bookkeeping, and models that combine features arithmetically inherit that accident.

Why each model family cares (or not)

Distance-based
kNN, k-means, RBF-SVM and DBSCAN compute ∥x−x′∥\lVert x-x'\rVert. The feature with the largest numeric range decides who is a neighbour; the rest are noise.
Variance-based
PCA finds directions of maximal variance. Area in m² has variance ≈ 4000; rooms ≈ 1.7. The first component is "area", whatever the correlations say.
Gradient-based
Linear and logistic regression, neural nets. Unequal scales stretch the loss surface into a narrow valley; gradient descent zig-zags across it and crawls along it.
Penalized
Ridge and lasso penalize ∥w∥\lVert w\rVert. A feature in small units needs a large weight and is penalized more, for no real reason.
Trees
A split asks is xj≤tx_j\le t?. Rescaling xjx_j just moves tt: the same rows go left. Trees are invariant to monotonic transforms, so scaling changes nothing.

Encoding has the opposite problem. "Paris", "Lyon" and "Nice" are not numbers, and most models (all of scikit-learn's, except a few) only accept numbers. Writing Lyon = 0, Nice = 1, Paris = 2 makes the data numeric, but a linear model then believes Paris is "two Nices" and that Nice lies between Lyon and Paris. A good encoding gives the model numbers without inventing an order or a distance the categories don't have.

The scalers

All the common scalers are affine: x′=(x−c)/sx' = (x - c)/s with a centre cc and scale ss computed per column on the training set. They differ only in which statistics they use, and that decides how they react to outliers.

z=x−μσ,μ=1n∑ixi,σ=1n∑i(xi−μ)2z=\frac{x-\mu}{\sigma},\qquad \mu=\frac1n\sum_{i}x_i,\quad \sigma=\sqrt{\frac1n\sum_i (x_i-\mu)^2}
zz
standardized value (a z-score): how many standard deviations x is from the mean
μ\mu
column mean on the training set
σ\sigma
column standard deviation (scikit-learn uses the population form, dividing by n)

Standardization (StandardScaler) gives every column mean 0 and variance 1. It is the default for linear models, SVMs, PCA and neural nets. It does not make data Gaussian and has no fixed range; since μ\mu and σ\sigma are both pulled by outliers, one extreme value shrinks everyone else's z-scores.

x′=x−min⁡(x)max⁡(x)−min⁡(x)∈[0,1]x'=\frac{x-\min(x)}{\max(x)-\min(x)}\in[0,1]
min⁡(x), max⁡(x)\min(x),\ \max(x)
smallest and largest training values of the column
x′x'
min-max scaled value; 0 at the training minimum, 1 at the maximum

Min-max scaling (MinMaxScaler) maps the training range onto [0,1][0,1]. It is handy when a model or a pixel-like input expects a bounded range, but it is the most fragile scaler: the range is set by the two most extreme points, so a single outlier compresses every other value into a sliver near 0. New data outside the training range lands outside [0,1][0,1].

x′=x−median⁡(x)Q3(x)−Q1(x)x'=\frac{x-\operatorname{median}(x)}{Q_3(x)-Q_1(x)}
median⁡(x)\operatorname{median}(x)
50th percentile of the training column
Q1, Q3Q_1,\ Q_3
25th and 75th percentiles; their difference is the interquartile range (IQR)

Robust scaling (RobustScaler) uses statistics that ignore the tails. Moving the largest value from 240 to 600 m² does not change the median or the quartiles at all, so the inliers keep their scaled values. The outlier itself is not tamed: it simply ends up far away, where it belongs.

x′=xmax⁡i∣xi∣∈[−1,1]x'=\frac{x}{\max_i |x_i|}\in[-1,1]
max⁡i∣xi∣\max_i|x_i|
largest absolute training value; no centring

Max-abs scaling (MaxAbsScaler) only divides. Because it never subtracts a centre, zeros stay zeros, so it is the scaler for sparse matrices (bag-of-words counts, one-hot blocks) where centring would turn millions of zeros into non-zeros. It is as outlier-sensitive as min-max.

Fixing the shape, not just the scale

Affine scalers change units, never shape: a right-skewed column (incomes, prices, counts) stays skewed, with a long tail that still dominates distances and squared errors. A monotonic non-linear transform compresses the tail first. Trees don't care about these either; linear models and kNN often do.

x(λ)={xλ−1λ,λ≠0ln⁡x,λ=0(x>0)x^{(\lambda)}=\begin{cases}\dfrac{x^{\lambda}-1}{\lambda}, & \lambda\neq 0\\[4pt] \ln x, & \lambda=0\end{cases}\qquad (x>0)
λ\lambda
power parameter, fitted by maximum likelihood so the result is as Gaussian as possible
ln⁡x\ln x
the λ→0\lambda\to0 limit: the plain log transform

Log (or log⁡(1+x)\log(1+x) for counts with zeros) is the everyday choice for multiplicative quantities: it turns ratios into differences. Box-Cox (above) learns the best power, but needs strictly positive data. Yeo-Johnson extends it to zero and negative values by applying a Box-Cox-like power to x+1x+1 for x≥0x\ge0 and to 1−x1-x (with power 2−λ2-\lambda) for x<0x<0. scikit-learn's PowerTransformer fits either and standardizes the result; QuantileTransformer goes further and maps each column through its empirical CDF onto a uniform or normal distribution.

Why gradient descent needs scaled features

For least-squares regression L(w)=12n∥Xw−y∥2L(w)=\frac{1}{2n}\lVert Xw-y\rVert^2, the loss is a quadratic bowl whose curvature is the Hessian:

∇2L=H=1nX⊤X,κ(H)=λmax⁡(H)λmin⁡(H)\nabla^2 L = H=\tfrac1n X^\top X,\qquad \kappa(H)=\frac{\lambda_{\max}(H)}{\lambda_{\min}(H)}
HH
Hessian: curvature of the loss; constant for a quadratic
λmax⁡, λmin⁡\lambda_{\max},\ \lambda_{\min}
largest and smallest eigenvalues: curvature along the steepest and flattest directions
κ\kappa
condition number: how elongated the elliptical loss contours are (axis ratio √κ)

If one column is measured in large units its variance is huge, so HH has one enormous eigenvalue and the contours become a long, thin valley. Gradient descent must keep the step below 2/λmax⁡2/\lambda_{\max} or it diverges along the steep direction, and with such a small step progress along the flat direction is glacial. Even with the best fixed step, the error shrinks per iteration by:

∥wt−w⋆∥≤(κ−1κ+1)t∥w0−w⋆∥⇒t≈κ2ln⁡1ε iterations\lVert w_{t}-w^\star\rVert \le \left(\frac{\kappa-1}{\kappa+1}\right)^{t}\lVert w_0-w^\star\rVert \quad\Rightarrow\quad t\approx\frac{\kappa}{2}\ln\frac{1}{\varepsilon}\ \text{iterations}
w⋆w^\star
the least-squares optimum
tt
iteration count
ε\varepsilon
target relative error

Iterations grow linearly in κ. Standardizing divides each column by its σ, which rescales HH to the correlation matrix of the features: all diagonal entries become 1 and the only elongation left comes from genuine correlation between features. Centring also matters when there is an intercept: an uncentred column of values near 100 is nearly parallel to the all-ones column, which is another source of ill-conditioning. (Normal equations and second-order methods are much less affected, but finite-precision arithmetic still prefers well-conditioned matrices.)

Worked example: six houses

Six houses with two numeric features and one categorical feature, plus a price we will use for target encoding. The query house Q has 80 m² and 1 room; we want its nearest neighbour.

HouseArea (m²)RoomsCityPrice (k€)
A451Paris300
B602Lyon200
C753Paris390
D903Lyon240
E1104Paris360
F2405Nice520
Q (query)801??
The training data. F is a large house that acts as an outlier in area.

Fitting each scaler by hand (training rows A–F only)

  1. 1Standard. Area: μ=620/6=103.33\mu=620/6=103.33; squared deviations sum to 24,983.324{,}983.3, so σ=24983.3/6=64.53\sigma=\sqrt{24983.3/6}=64.53. Rooms: μ=3\mu=3, σ=10/6=1.291\sigma=\sqrt{10/6}=1.291. House A: z=((45−103.33)/64.53, (1−3)/1.291)=(−0.904, −1.549)z=\big((45-103.33)/64.53,\ (1-3)/1.291\big)=(-0.904,\ -1.549).
  2. 2Min-max. Area: min 45, max 240, range 195. Rooms: min 1, range 4. A → (0,0)(0, 0), E → (65/195, 3/4)=(0.333, 0.75)(65/195,\ 3/4)=(0.333,\ 0.75), F → (1,1)(1, 1).
  3. 3Robust. Sorted areas 45, 60, 75, 90, 110, 240: median 82.582.5, Q1=63.75Q_1=63.75, Q3=105Q_3=105 (linear interpolation), IQR 41.2541.25. Rooms: median 3, Q1=2.25Q_1=2.25, Q3=3.75Q_3=3.75, IQR 1.51.5. A → (−0.909, −1.333)(-0.909,\ -1.333); F → (3.818, 1.333)(3.818,\ 1.333).
  4. 4Max-abs. Divide by 240 and 5: A → (0.188, 0.2)(0.188,\ 0.2), F → (1,1)(1, 1).
  5. 5Transform the query with the same numbers: Q → standard (−0.362, −1.549)(-0.362,\ -1.549), min-max (0.179, 0)(0.179,\ 0), robust (−0.061, −1.333)(-0.061,\ -1.333), max-abs (0.333, 0.2)(0.333,\ 0.2). Q is never used to fit anything.
HouseStandardMin-maxRobustMax-abs
A(−0.904, −1.549)(0.000, 0.00)(−0.909, −1.333)(0.188, 0.2)
B(−0.672, −0.775)(0.077, 0.25)(−0.545, −0.667)(0.250, 0.4)
C(−0.439, 0.000)(0.154, 0.50)(−0.182, 0.000)(0.313, 0.6)
D(−0.207, 0.000)(0.231, 0.50)(0.182, 0.000)(0.375, 0.6)
E(0.103, 0.775)(0.333, 0.75)(0.667, 0.667)(0.458, 0.8)
F(2.118, 1.549)(1.000, 1.00)(3.818, 1.333)(1.000, 1.0)
Q(−0.362, −1.549)(0.179, 0.00)(−0.061, −1.333)(0.333, 0.2)
Scaled coordinates (area′, rooms′) of every house.
HouseRawStandardMin-maxRobustMax-abs
A35.0000.5420.1790.8480.146
B20.0250.8340.2700.8240.217
C5.3851.5510.5011.3390.401
D10.1981.5570.5031.3550.402
E30.1502.3700.7662.1280.613
F160.0503.9681.2944.7071.041
Euclidean distance from Q to each house. Bold = nearest neighbour. Without scaling, rooms are invisible.

On raw features Q's nearest neighbour is C (75 m², 3 rooms): the 5 m² gap in area is all that counts, and the 2-room gap adds only 25+4−5=0.39\sqrt{25+4}-5=0.39 to the distance. After standardization a 2-room difference is worth as much as a 100 m² difference (2/1.291=1.5492/1.291=1.549 standard deviations either way), and the nearest neighbour becomes A (45 m², 1 room, the only other one-room house). Min-max and max-abs agree. Robust scaling divides area by its IQR (41.25) rather than σ (64.53), so area counts relatively more and B (20 m² and 1 room away) edges out A (35 m² away, same rooms): 0.824 vs 0.848. A 1-NN price prediction for Q changes from 390 k€ to 300 k€ (or 200 k€). The choice of scaler is a modelling decision: it decides how many m² one room is worth.

Now make F a real outlier, 600 m². Min-max squeezes houses A–E into area′ ∈[0, 0.117]\in[0,\ 0.117] (down from [0, 0.333][0,\ 0.333]), so area barely matters any more. Standardization shrinks their spread from 1.007 to 0.331 because σ\sigma grows. Robust scaling leaves A–E exactly where they were (spread 1.576) and sends F to 12.5: the median and quartiles never looked at F.

F area240
area′ (x) vs rooms′ (y) · equal axis scaleABCDEFQ050100150200250300rooms 1–5 spans 3.5 px here; area spans 171 pxno scalingx′ = xquery Q = (80 m², 1)nearest: C (d = 5.385)A–E area′ spread:65 m²● Paris ● Lyon● Nice ◆ query▸ = off-plot, value shown◂ raw area (m²)drag F: the outlier0100200300400500600F 240
housearearoomsarea′rooms′dist to Q
A451——35.000
B602——20.025
C753——5.385 ← nearest
D903——10.198
E1104——30.150
F2405——160.050
fig.Three views. Scalers: the six houses with each scaler applied, drawn with equal axis scale so the dashed circle really is the set of points at Q's nearest-neighbour distance. Drag F along the raw-area strip and watch min-max crush A–E while robust scaling leaves them alone. Gradient descent: one quadratic loss on raw features (spreads 4 and 1, correlation 0.25, κ = 17.2) and the same loss after standardizing (κ = 1.67), from the same starting model; the raw path zig-zags across the valley and needs 41 steps where the standardized one needs 9 (step 1.7/λmax). Encoding: the city column under each encoder, with a slider for the target-encoding smoothing m.

Encoding categories

onehot(c)=(1[c=v1], 1[c=v2], …, 1[c=vK])∈{0,1}K\text{onehot}(c)=\big(\mathbb 1[c=v_1],\ \mathbb 1[c=v_2],\ \dots,\ \mathbb 1[c=v_K]\big)\in\{0,1\}^K
v1,…,vKv_1,\dots,v_K
the K categories seen in training
1[⋅]\mathbb 1[\cdot]
indicator: 1 if true, else 0; exactly one entry is 1

One-hot encoding gives each category its own 0/1 column. All categories are equidistant (any two differ in exactly two coordinates, distance 2\sqrt2), so no order is invented. A linear model learns one coefficient per category: a per-city offset. Cost: KK columns, which is fine for 3 cities and painful for 30,000 ZIP codes.

Ordinal encoding maps categories to integers 0,1,…,K−10,1,\dots,K-1. It is right when the categories are ordered (small < medium < large; bad < ok < good) and you choose the order. For unordered categories it is fine for trees, which can isolate any code with two splits, and harmful for linear models and kNN, which read the codes as quantities.

TE⁡(c)=nc yˉc+m yˉnc+m=∑i: xi=cyi+m yˉnc+m\operatorname{TE}(c)=\frac{n_c\,\bar y_c+m\,\bar y}{n_c+m}=\frac{\sum_{i:\,x_i=c}y_i+m\,\bar y}{n_c+m}
ncn_c
number of training rows in category c
yˉc\bar y_c
mean target among those rows
yˉ\bar y
global mean target: the prior
mm
smoothing strength: the prior counts as m pseudo-rows

Target (mean) encoding replaces each category by the average target of its rows: one dense column, however many categories. The m-estimate above shrinks small categories toward the global mean: with nc≫mn_c\gg m the category speaks for itself; with nc≪mn_c\ll m it is mostly the prior. It is a Bayesian posterior mean under a prior worth mm observations. For classification, yy is the 0/1 label and TE⁡\operatorname{TE} a smoothed class rate.

Frequency (count) encoding replaces a category by its share (or count) of training rows. It needs no target, so it cannot leak, and it captures "popular vs rare", but two categories with the same count collide. Hashing maps each category through a hash function into one of DD buckets, with no stored vocabulary at all:

ϕ(c)j=∑c : h(c)=js(c),h:strings→{0,…,D−1},  s(c)∈{−1,+1}\phi(c)_j=\sum_{c\,:\,h(c)=j}s(c),\qquad h:\text{strings}\to\{0,\dots,D-1\},\ \ s(c)\in\{-1,+1\}
DD
number of hash buckets (output columns), chosen by you, e.g. 2¹⁸
hh
hash function deciding the bucket
ss
second hash giving a random sign, so collisions cancel on average instead of adding up

Hashing (FeatureHasher) handles unseen and unbounded vocabularies (user IDs, URLs, word n-grams) in fixed memory, at the price of collisions and unreadable columns. Neural networks usually go one step further and learn an embedding vector per category.

Worked example: encoding the city

HouseCityOne-hot (L, N, P)Drop-first (N, P)OrdinalFrequency
AParis(0, 0, 1)(0, 1)20.500
BLyon(1, 0, 0)(0, 0)00.333
CParis(0, 0, 1)(0, 1)20.500
DLyon(1, 0, 0)(0, 0)00.333
EParis(0, 0, 1)(0, 1)20.500
FNice(0, 1, 0)(1, 0)10.167
City under four encoders. One-hot uses the alphabetical order Lyon, Nice, Paris; drop-first removes Lyon, the reference.

Target encoding with smoothing m = 2

  1. 1Global mean price: yˉ=(300+200+390+240+360+520)/6=2010/6=335\bar y=(300+200+390+240+360+520)/6=2010/6=335.
  2. 2Paris (A, C, E): n=3n=3, sum 10501050, raw mean 350350. TE⁡=(1050+2⋅335)/(3+2)=1720/5=344.0\operatorname{TE}=(1050+2\cdot335)/(3+2)=1720/5=344.0.
  3. 3Lyon (B, D): n=2n=2, sum 440440, raw mean 220220. TE⁡=(440+670)/4=277.5\operatorname{TE}=(440+670)/4=277.5.
  4. 4Nice (F only): n=1n=1, raw mean 520520. TE⁡=(520+670)/3=396.7\operatorname{TE}=(520+670)/3=396.7. Without smoothing (m=0m=0) Nice's code would be 520, which is F's own price: a perfect leak.
  5. 5Smoothing moves each code toward 335 in proportion to m/(nc+m)m/(n_c+m): Paris by 40%, Lyon by 50%, Nice by 67%. The rarest category is trusted least.
  6. 6Cross-fitted instead (3 unshuffled folds {A,B}, {C,D}, {E,F}): F is encoded from A–D only. Nice never appears there, so F gets the fold's prior, 282.5282.5, and its own price cannot leak.

In code

python
1import numpy as np
2import pandas as pd
3from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, OneHotEncoder, TargetEncoder
4
5houses = pd.DataFrame({
6 "area": [45, 60, 75, 90, 110, 240],
7 "rooms": [1, 2, 3, 3, 4, 5],
8 "city": ["Paris", "Lyon", "Paris", "Lyon", "Paris", "Nice"],
9 "price": [300, 200, 390, 240, 360, 520],
10}, index=list("ABCDEF"))
11X, q = houses[["area", "rooms"]], pd.DataFrame({"area": [80], "rooms": [1]})
12
13for scaler in [None, StandardScaler(), MinMaxScaler(), RobustScaler()]:
14 Z, Zq = (X.to_numpy(), q.to_numpy()) if scaler is None else (scaler.fit_transform(X), scaler.transform(q))
15 d = np.linalg.norm(Z - Zq, axis=1)
16 print(f"{type(scaler).__name__:15s} nearest = {houses.index[d.argmin()]} d = {d.round(3)}")
17# NoneType nearest = C d = [ 35. 20.025 5.385 10.198 30.15 160.05 ]
18# StandardScaler nearest = A d = [0.542 0.834 1.551 1.557 2.37 3.968]
19# MinMaxScaler nearest = A d = [0.179 0.27 0.501 0.503 0.766 1.294]
20# RobustScaler nearest = B d = [0.848 0.824 1.339 1.355 2.128 4.707]
21
22print(OneHotEncoder(drop="first").fit(houses[["city"]]).get_feature_names_out()) # ['city_Nice' 'city_Paris']
23te = TargetEncoder(smooth=2.0, target_type="continuous").fit(houses[["city"]], houses.price)
24print(dict(zip(te.categories_[0], te.encodings_[0].round(2)))) # Lyon 277.5, Nice 396.67, Paris 344.0
Reproducing the worked example with scikit-learn. The printed numbers match the tables above.
python
1import numpy as np
2import pandas as pd
3from sklearn.compose import ColumnTransformer
4from sklearn.pipeline import Pipeline
5from sklearn.preprocessing import StandardScaler, OneHotEncoder
6from sklearn.neighbors import KNeighborsRegressor
7from sklearn.model_selection import cross_val_score
8
9rng = np.random.default_rng(0)
10n = 600
11df = pd.DataFrame({
12 "area": rng.lognormal(4.4, 0.4, n).round(), # m², roughly 40-250
13 "rooms": rng.integers(1, 7, n),
14 "city": rng.choice(["Paris", "Lyon", "Nice", "Lille"], n, p=[0.4, 0.3, 0.2, 0.1]),
15})
16df["price"] = (2.5 * df.area + 15 * df.rooms
17 + df.city.map({"Paris": 150, "Lyon": 40, "Nice": 90, "Lille": 0})
18 + rng.normal(0, 25, n))
19X, y = df.drop(columns="price"), df.price
20
21def knn(numeric):
22 pre = ColumnTransformer([
23 ("num", numeric, ["area", "rooms"]),
24 ("cat", OneHotEncoder(handle_unknown="ignore"), ["city"]),
25 ])
26 return Pipeline([("pre", pre), ("knn", KNeighborsRegressor(n_neighbors=10))])
27
28# cross_val_score re-fits the scaler and encoder on each training fold: no leakage.
29print("raw R2:", cross_val_score(knn("passthrough"), X, y, cv=5).mean().round(3)) # 0.709
30print("scaled R2:", cross_val_score(knn(StandardScaler()), X, y, cv=5).mean().round(3)) # 0.903
31
32model = knn(StandardScaler()).fit(X, y)
33new = pd.DataFrame({"area": [80], "rooms": [3], "city": ["Marseille"]}) # a city never seen in training
34print(model.predict(new)) # works: an unknown city becomes an all-zero one-hot block
The production pattern: ColumnTransformer inside a Pipeline, so every preprocessing statistic is fitted on training folds only.
python
1import numpy as np, pandas as pd
2from sklearn.linear_model import Ridge
3from sklearn.pipeline import make_pipeline
4from sklearn.preprocessing import TargetEncoder
5from sklearn.model_selection import cross_val_score
6
7rng = np.random.default_rng(1)
8zips = rng.integers(0, 500, 1000).astype(str) # 500 categories, ~2 rows each
9y = rng.normal(0, 1, 1000) # no relation to zip at all
10
11# Leaky: category means computed on ALL rows, before cross-validation
12leaky = pd.DataFrame({"zip_te": pd.Series(zips).map(pd.Series(y).groupby(zips).mean())})
13print(cross_val_score(Ridge(), leaky, y, cv=5).mean().round(3)) # 0.45 (fake)
14
15# Correct: TargetEncoder inside the pipeline, cross-fitted on each training fold
16good = make_pipeline(TargetEncoder(target_type="continuous"), Ridge())
17print(cross_val_score(good, pd.DataFrame({"zip": zips}), y, cv=5).mean().round(3)) # -0.008
Why naive target encoding is dangerous: on a target that is pure noise it fakes a strong signal.

Which scaler, which encoder?

SituationUseWhy
Linear / logistic models, SVM, PCA, neural netsStandardScalerzero mean, unit variance: well-conditioned gradients and fair penalties
Heavy outliers you want to keepRobustScalermedian and IQR ignore the tails, so inliers keep their spread
Bounded inputs expected (images, some activations)MinMaxScalerexact [0, 1] range; only if there are no wild outliers
Sparse data (text counts, one-hot blocks)MaxAbsScaler (or none)no centring keeps zeros zero
Right-skewed positive data (income, price, counts)log / log1p, Box-Cox, then standardizecompress the tail before choosing a scale
Trees, random forests, gradient boostingno scalingsplits are invariant to monotonic transforms
Few unordered categories (< ~15)OneHotEncoderno invented order; drop='first' only for unregularized linear models
Genuinely ordered categoriesOrdinalEncoder with an explicit orderkeeps the order in one column
High cardinality (ZIP, product ID) with a targetTargetEncoder (cross-fitted, smoothed)one dense column; smoothing tames rare categories
Huge or open vocabulary, streamingFeatureHasherfixed memory, no vocabulary, handles unseen values
Tree models with categoriesnative categorical support (LightGBM, CatBoost, HistGradientBoosting) or ordinal codestrees split on codes directly
Rules of thumb. When in doubt, cross-validate the choice: it is a hyperparameter like any other.

Use when

  • +Scale when the model uses distances, dot products, variances, gradients or norm penalties: kNN, k-means, SVM, PCA, linear models, neural nets.
  • +Standardization makes gradient descent converge in tens of steps instead of hundreds of thousands, and makes coefficient sizes comparable.
  • +One-hot is the safe default for low-cardinality nominal features: no assumptions, interpretable coefficients.
  • +Target encoding compresses thousands of categories into one informative column, which is often the best feature in tabular data.

Avoid when

  • −Skip scaling for tree ensembles: it cannot help, and it makes the stored thresholds harder to read.
  • −Scaling makes features comparable in units, not in importance: it weights every feature equally, which may drown a crucial feature among noisy ones.
  • −One-hot explodes with high cardinality and creates rare, noisy columns; hashing and target encoding trade interpretability for size.
  • −Target encoding needs cross-fitting and smoothing; done naively it leaks the label and produces optimistic validation scores.

Check yourself

Which of these models gives exactly the same predictions whether or not you standardize the numeric features?

Check yourself

House F's area grows from 240 to 600 m² while A–E stay the same. Which scaler leaves the scaled areas of A–E unchanged?

Check yourself

Nice has one house, priced 520 k€; the global mean price is 335 k€. What is Nice's target encoding with smoothing m=2m=2?

Check yourself

You one-hot encode a 4-category feature into 4 columns and fit ordinary least squares with an intercept. What goes wrong?

Remember

  • ✓Models that use distances, variances, dot products, gradients or norm penalties are sensitive to units; trees are not.
  • ✓Standardize (x−μ)/σ(x-\mu)/\sigma by default; min-max for bounded inputs without outliers; robust (median/IQR) when outliers would squash everyone else; max-abs for sparse data; log/Box-Cox/Yeo-Johnson to fix skew.
  • ✓Unscaled features make X⊤XX^\top X ill-conditioned: elongated contours, zig-zagging gradient descent, iterations ∝ κ. Six houses: κ 131,766 → 15.7 and 668,829 → 59 iterations.
  • ✓Scaling changes who your nearest neighbours are, so the scaler is a modelling choice: in the example Q's neighbour moved from C to A (or B).
  • ✓One-hot for few nominal categories (drop one only for unregularized linear models); ordinal only for truly ordered ones; target encoding with smoothing and cross-fitting for high cardinality; hashing for open vocabularies.
  • ✓Fit every scaler and encoder on training data only, inside a Pipeline; plan for unseen categories.
← back to ML Algorithms roadmap