Definition
Feature scaling re-expresses each numeric column in comparable units, typically by subtracting a centre and dividing by a spread, so that no feature dominates a distance, a gradient or a penalty just because of the units it was recorded in.
Definition
Categorical encoding turns a non-numeric column (city, colour, product ID) into one or more numeric columns that a model can do arithmetic on, without inventing an order or a distance the data does not have.
At a glance
- Stage
- preprocessing: fitted on the training data, then applied unchanged to validation, test and production data
- Learns
- per column: a centre and a scale (mean/σ, min/range, median/IQR), or a category → code mapping
- Who needs scaling
- kNN, k-means, SVM (RBF), PCA, linear/logistic regression trained by gradient descent, ridge/lasso, neural networks
- Who doesn't
- decision trees, random forests, gradient-boosted trees: a split threshold is unaffected by any monotonic transform
- Main encoders
- one-hot, ordinal, target (mean) encoding with smoothing, frequency, hashing
- Main risk
- leakage: fitting the scaler or target encoder on rows you later evaluate on
The problem
A house has an area of 80 m² and 1 room. Another has 75 m² and 3 rooms. Which one is "closer" to a third house? A Euclidean distance adds , so a 5 m² difference in area counts as much as a 5-room difference. The answer depends on whether you measured area in m², ft² or km², which is absurd. Units are an accident of bookkeeping, and models that combine features arithmetically inherit that accident.
Why each model family cares (or not)
- Distance-based
- kNN, k-means, RBF-SVM and DBSCAN compute . The feature with the largest numeric range decides who is a neighbour; the rest are noise.
- Variance-based
- PCA finds directions of maximal variance. Area in m² has variance ≈ 4000; rooms ≈ 1.7. The first component is "area", whatever the correlations say.
- Gradient-based
- Linear and logistic regression, neural nets. Unequal scales stretch the loss surface into a narrow valley; gradient descent zig-zags across it and crawls along it.
- Penalized
- Ridge and lasso penalize . A feature in small units needs a large weight and is penalized more, for no real reason.
- Trees
- A split asks is ?. Rescaling just moves : the same rows go left. Trees are invariant to monotonic transforms, so scaling changes nothing.
Encoding has the opposite problem. "Paris", "Lyon" and "Nice" are not numbers, and most models (all of scikit-learn's, except a few) only accept numbers. Writing Lyon = 0, Nice = 1, Paris = 2 makes the data numeric, but a linear model then believes Paris is "two Nices" and that Nice lies between Lyon and Paris. A good encoding gives the model numbers without inventing an order or a distance the categories don't have.
The scalers
All the common scalers are affine: with a centre and scale computed per column on the training set. They differ only in which statistics they use, and that decides how they react to outliers.
- standardized value (a z-score): how many standard deviations x is from the mean
- column mean on the training set
- column standard deviation (scikit-learn uses the population form, dividing by n)
Standardization (StandardScaler) gives every column mean 0 and variance 1. It is the default for linear models, SVMs, PCA and neural nets. It does not make data Gaussian and has no fixed range; since and are both pulled by outliers, one extreme value shrinks everyone else's z-scores.
- smallest and largest training values of the column
- min-max scaled value; 0 at the training minimum, 1 at the maximum
Min-max scaling (MinMaxScaler) maps the training range onto . It is handy when a model or a pixel-like input expects a bounded range, but it is the most fragile scaler: the range is set by the two most extreme points, so a single outlier compresses every other value into a sliver near 0. New data outside the training range lands outside .
- 50th percentile of the training column
- 25th and 75th percentiles; their difference is the interquartile range (IQR)
Robust scaling (RobustScaler) uses statistics that ignore the tails. Moving the largest value from 240 to 600 m² does not change the median or the quartiles at all, so the inliers keep their scaled values. The outlier itself is not tamed: it simply ends up far away, where it belongs.
- largest absolute training value; no centring
Max-abs scaling (MaxAbsScaler) only divides. Because it never subtracts a centre, zeros stay zeros, so it is the scaler for sparse matrices (bag-of-words counts, one-hot blocks) where centring would turn millions of zeros into non-zeros. It is as outlier-sensitive as min-max.
Fixing the shape, not just the scale
Affine scalers change units, never shape: a right-skewed column (incomes, prices, counts) stays skewed, with a long tail that still dominates distances and squared errors. A monotonic non-linear transform compresses the tail first. Trees don't care about these either; linear models and kNN often do.
- power parameter, fitted by maximum likelihood so the result is as Gaussian as possible
- the limit: the plain log transform
Log (or for counts with zeros) is the everyday choice for multiplicative quantities: it turns ratios into differences. Box-Cox (above) learns the best power, but needs strictly positive data. Yeo-Johnson extends it to zero and negative values by applying a Box-Cox-like power to for and to (with power ) for . scikit-learn's PowerTransformer fits either and standardizes the result; QuantileTransformer goes further and maps each column through its empirical CDF onto a uniform or normal distribution.
Why gradient descent needs scaled features
For least-squares regression , the loss is a quadratic bowl whose curvature is the Hessian:
- Hessian: curvature of the loss; constant for a quadratic
- largest and smallest eigenvalues: curvature along the steepest and flattest directions
- condition number: how elongated the elliptical loss contours are (axis ratio √κ)
If one column is measured in large units its variance is huge, so has one enormous eigenvalue and the contours become a long, thin valley. Gradient descent must keep the step below or it diverges along the steep direction, and with such a small step progress along the flat direction is glacial. Even with the best fixed step, the error shrinks per iteration by:
- the least-squares optimum
- iteration count
- target relative error
Iterations grow linearly in κ. Standardizing divides each column by its σ, which rescales to the correlation matrix of the features: all diagonal entries become 1 and the only elongation left comes from genuine correlation between features. Centring also matters when there is an intercept: an uncentred column of values near 100 is nearly parallel to the all-ones column, which is another source of ill-conditioning. (Normal equations and second-order methods are much less affected, but finite-precision arithmetic still prefers well-conditioned matrices.)
Worked example: six houses
Six houses with two numeric features and one categorical feature, plus a price we will use for target encoding. The query house Q has 80 m² and 1 room; we want its nearest neighbour.
| House | Area (m²) | Rooms | City | Price (k€) |
|---|---|---|---|---|
| A | 45 | 1 | Paris | 300 |
| B | 60 | 2 | Lyon | 200 |
| C | 75 | 3 | Paris | 390 |
| D | 90 | 3 | Lyon | 240 |
| E | 110 | 4 | Paris | 360 |
| F | 240 | 5 | Nice | 520 |
| Q (query) | 80 | 1 | ? | ? |
Fitting each scaler by hand (training rows A–F only)
- 1Standard. Area: ; squared deviations sum to , so . Rooms: , . House A: .
- 2Min-max. Area: min 45, max 240, range 195. Rooms: min 1, range 4. A → , E → , F → .
- 3Robust. Sorted areas 45, 60, 75, 90, 110, 240: median , , (linear interpolation), IQR . Rooms: median 3, , , IQR . A → ; F → .
- 4Max-abs. Divide by 240 and 5: A → , F → .
- 5Transform the query with the same numbers: Q → standard , min-max , robust , max-abs . Q is never used to fit anything.
| House | Standard | Min-max | Robust | Max-abs |
|---|---|---|---|---|
| A | (−0.904, −1.549) | (0.000, 0.00) | (−0.909, −1.333) | (0.188, 0.2) |
| B | (−0.672, −0.775) | (0.077, 0.25) | (−0.545, −0.667) | (0.250, 0.4) |
| C | (−0.439, 0.000) | (0.154, 0.50) | (−0.182, 0.000) | (0.313, 0.6) |
| D | (−0.207, 0.000) | (0.231, 0.50) | (0.182, 0.000) | (0.375, 0.6) |
| E | (0.103, 0.775) | (0.333, 0.75) | (0.667, 0.667) | (0.458, 0.8) |
| F | (2.118, 1.549) | (1.000, 1.00) | (3.818, 1.333) | (1.000, 1.0) |
| Q | (−0.362, −1.549) | (0.179, 0.00) | (−0.061, −1.333) | (0.333, 0.2) |
| House | Raw | Standard | Min-max | Robust | Max-abs |
|---|---|---|---|---|---|
| A | 35.000 | 0.542 | 0.179 | 0.848 | 0.146 |
| B | 20.025 | 0.834 | 0.270 | 0.824 | 0.217 |
| C | 5.385 | 1.551 | 0.501 | 1.339 | 0.401 |
| D | 10.198 | 1.557 | 0.503 | 1.355 | 0.402 |
| E | 30.150 | 2.370 | 0.766 | 2.128 | 0.613 |
| F | 160.050 | 3.968 | 1.294 | 4.707 | 1.041 |
On raw features Q's nearest neighbour is C (75 m², 3 rooms): the 5 m² gap in area is all that counts, and the 2-room gap adds only to the distance. After standardization a 2-room difference is worth as much as a 100 m² difference ( standard deviations either way), and the nearest neighbour becomes A (45 m², 1 room, the only other one-room house). Min-max and max-abs agree. Robust scaling divides area by its IQR (41.25) rather than σ (64.53), so area counts relatively more and B (20 m² and 1 room away) edges out A (35 m² away, same rooms): 0.824 vs 0.848. A 1-NN price prediction for Q changes from 390 k€ to 300 k€ (or 200 k€). The choice of scaler is a modelling decision: it decides how many m² one room is worth.
Now make F a real outlier, 600 m². Min-max squeezes houses A–E into area′ (down from ), so area barely matters any more. Standardization shrinks their spread from 1.007 to 0.331 because grows. Robust scaling leaves A–E exactly where they were (spread 1.576) and sends F to 12.5: the median and quartiles never looked at F.
| house | area | rooms | area′ | rooms′ | dist to Q |
|---|---|---|---|---|---|
| A | 45 | 1 | — | — | 35.000 |
| B | 60 | 2 | — | — | 20.025 |
| C | 75 | 3 | — | — | 5.385 ← nearest |
| D | 90 | 3 | — | — | 10.198 |
| E | 110 | 4 | — | — | 30.150 |
| F | 240 | 5 | — | — | 160.050 |
Encoding categories
- the K categories seen in training
- indicator: 1 if true, else 0; exactly one entry is 1
One-hot encoding gives each category its own 0/1 column. All categories are equidistant (any two differ in exactly two coordinates, distance ), so no order is invented. A linear model learns one coefficient per category: a per-city offset. Cost: columns, which is fine for 3 cities and painful for 30,000 ZIP codes.
Ordinal encoding maps categories to integers . It is right when the categories are ordered (small < medium < large; bad < ok < good) and you choose the order. For unordered categories it is fine for trees, which can isolate any code with two splits, and harmful for linear models and kNN, which read the codes as quantities.
- number of training rows in category c
- mean target among those rows
- global mean target: the prior
- smoothing strength: the prior counts as m pseudo-rows
Target (mean) encoding replaces each category by the average target of its rows: one dense column, however many categories. The m-estimate above shrinks small categories toward the global mean: with the category speaks for itself; with it is mostly the prior. It is a Bayesian posterior mean under a prior worth observations. For classification, is the 0/1 label and a smoothed class rate.
Frequency (count) encoding replaces a category by its share (or count) of training rows. It needs no target, so it cannot leak, and it captures "popular vs rare", but two categories with the same count collide. Hashing maps each category through a hash function into one of buckets, with no stored vocabulary at all:
- number of hash buckets (output columns), chosen by you, e.g. 2¹⁸
- hash function deciding the bucket
- second hash giving a random sign, so collisions cancel on average instead of adding up
Hashing (FeatureHasher) handles unseen and unbounded vocabularies (user IDs, URLs, word n-grams) in fixed memory, at the price of collisions and unreadable columns. Neural networks usually go one step further and learn an embedding vector per category.
Worked example: encoding the city
| House | City | One-hot (L, N, P) | Drop-first (N, P) | Ordinal | Frequency |
|---|---|---|---|---|---|
| A | Paris | (0, 0, 1) | (0, 1) | 2 | 0.500 |
| B | Lyon | (1, 0, 0) | (0, 0) | 0 | 0.333 |
| C | Paris | (0, 0, 1) | (0, 1) | 2 | 0.500 |
| D | Lyon | (1, 0, 0) | (0, 0) | 0 | 0.333 |
| E | Paris | (0, 0, 1) | (0, 1) | 2 | 0.500 |
| F | Nice | (0, 1, 0) | (1, 0) | 1 | 0.167 |
Target encoding with smoothing m = 2
- 1Global mean price: .
- 2Paris (A, C, E): , sum , raw mean . .
- 3Lyon (B, D): , sum , raw mean . .
- 4Nice (F only): , raw mean . . Without smoothing () Nice's code would be 520, which is F's own price: a perfect leak.
- 5Smoothing moves each code toward 335 in proportion to : Paris by 40%, Lyon by 50%, Nice by 67%. The rarest category is trusted least.
- 6Cross-fitted instead (3 unshuffled folds {A,B}, {C,D}, {E,F}): F is encoded from A–D only. Nice never appears there, so F gets the fold's prior, , and its own price cannot leak.
In code
1import numpy as np2import pandas as pd3from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, OneHotEncoder, TargetEncoder45houses = pd.DataFrame({6 "area": [45, 60, 75, 90, 110, 240],7 "rooms": [1, 2, 3, 3, 4, 5],8 "city": ["Paris", "Lyon", "Paris", "Lyon", "Paris", "Nice"],9 "price": [300, 200, 390, 240, 360, 520],10}, index=list("ABCDEF"))11X, q = houses[["area", "rooms"]], pd.DataFrame({"area": [80], "rooms": [1]})1213for scaler in [None, StandardScaler(), MinMaxScaler(), RobustScaler()]:14 Z, Zq = (X.to_numpy(), q.to_numpy()) if scaler is None else (scaler.fit_transform(X), scaler.transform(q))15 d = np.linalg.norm(Z - Zq, axis=1)16 print(f"{type(scaler).__name__:15s} nearest = {houses.index[d.argmin()]} d = {d.round(3)}")17# NoneType nearest = C d = [ 35. 20.025 5.385 10.198 30.15 160.05 ]18# StandardScaler nearest = A d = [0.542 0.834 1.551 1.557 2.37 3.968]19# MinMaxScaler nearest = A d = [0.179 0.27 0.501 0.503 0.766 1.294]20# RobustScaler nearest = B d = [0.848 0.824 1.339 1.355 2.128 4.707]2122print(OneHotEncoder(drop="first").fit(houses[["city"]]).get_feature_names_out()) # ['city_Nice' 'city_Paris']23te = TargetEncoder(smooth=2.0, target_type="continuous").fit(houses[["city"]], houses.price)24print(dict(zip(te.categories_[0], te.encodings_[0].round(2)))) # Lyon 277.5, Nice 396.67, Paris 344.01import numpy as np2import pandas as pd3from sklearn.compose import ColumnTransformer4from sklearn.pipeline import Pipeline5from sklearn.preprocessing import StandardScaler, OneHotEncoder6from sklearn.neighbors import KNeighborsRegressor7from sklearn.model_selection import cross_val_score89rng = np.random.default_rng(0)10n = 60011df = pd.DataFrame({12 "area": rng.lognormal(4.4, 0.4, n).round(), # m², roughly 40-25013 "rooms": rng.integers(1, 7, n),14 "city": rng.choice(["Paris", "Lyon", "Nice", "Lille"], n, p=[0.4, 0.3, 0.2, 0.1]),15})16df["price"] = (2.5 * df.area + 15 * df.rooms17 + df.city.map({"Paris": 150, "Lyon": 40, "Nice": 90, "Lille": 0})18 + rng.normal(0, 25, n))19X, y = df.drop(columns="price"), df.price2021def knn(numeric):22 pre = ColumnTransformer([23 ("num", numeric, ["area", "rooms"]),24 ("cat", OneHotEncoder(handle_unknown="ignore"), ["city"]),25 ])26 return Pipeline([("pre", pre), ("knn", KNeighborsRegressor(n_neighbors=10))])2728# cross_val_score re-fits the scaler and encoder on each training fold: no leakage.29print("raw R2:", cross_val_score(knn("passthrough"), X, y, cv=5).mean().round(3)) # 0.70930print("scaled R2:", cross_val_score(knn(StandardScaler()), X, y, cv=5).mean().round(3)) # 0.9033132model = knn(StandardScaler()).fit(X, y)33new = pd.DataFrame({"area": [80], "rooms": [3], "city": ["Marseille"]}) # a city never seen in training34print(model.predict(new)) # works: an unknown city becomes an all-zero one-hot block1import numpy as np, pandas as pd2from sklearn.linear_model import Ridge3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import TargetEncoder5from sklearn.model_selection import cross_val_score67rng = np.random.default_rng(1)8zips = rng.integers(0, 500, 1000).astype(str) # 500 categories, ~2 rows each9y = rng.normal(0, 1, 1000) # no relation to zip at all1011# Leaky: category means computed on ALL rows, before cross-validation12leaky = pd.DataFrame({"zip_te": pd.Series(zips).map(pd.Series(y).groupby(zips).mean())})13print(cross_val_score(Ridge(), leaky, y, cv=5).mean().round(3)) # 0.45 (fake)1415# Correct: TargetEncoder inside the pipeline, cross-fitted on each training fold16good = make_pipeline(TargetEncoder(target_type="continuous"), Ridge())17print(cross_val_score(good, pd.DataFrame({"zip": zips}), y, cv=5).mean().round(3)) # -0.008Which scaler, which encoder?
| Situation | Use | Why |
|---|---|---|
| Linear / logistic models, SVM, PCA, neural nets | StandardScaler | zero mean, unit variance: well-conditioned gradients and fair penalties |
| Heavy outliers you want to keep | RobustScaler | median and IQR ignore the tails, so inliers keep their spread |
| Bounded inputs expected (images, some activations) | MinMaxScaler | exact [0, 1] range; only if there are no wild outliers |
| Sparse data (text counts, one-hot blocks) | MaxAbsScaler (or none) | no centring keeps zeros zero |
| Right-skewed positive data (income, price, counts) | log / log1p, Box-Cox, then standardize | compress the tail before choosing a scale |
| Trees, random forests, gradient boosting | no scaling | splits are invariant to monotonic transforms |
| Few unordered categories (< ~15) | OneHotEncoder | no invented order; drop='first' only for unregularized linear models |
| Genuinely ordered categories | OrdinalEncoder with an explicit order | keeps the order in one column |
| High cardinality (ZIP, product ID) with a target | TargetEncoder (cross-fitted, smoothed) | one dense column; smoothing tames rare categories |
| Huge or open vocabulary, streaming | FeatureHasher | fixed memory, no vocabulary, handles unseen values |
| Tree models with categories | native categorical support (LightGBM, CatBoost, HistGradientBoosting) or ordinal codes | trees split on codes directly |
Use when
- +Scale when the model uses distances, dot products, variances, gradients or norm penalties: kNN, k-means, SVM, PCA, linear models, neural nets.
- +Standardization makes gradient descent converge in tens of steps instead of hundreds of thousands, and makes coefficient sizes comparable.
- +One-hot is the safe default for low-cardinality nominal features: no assumptions, interpretable coefficients.
- +Target encoding compresses thousands of categories into one informative column, which is often the best feature in tabular data.
Avoid when
- −Skip scaling for tree ensembles: it cannot help, and it makes the stored thresholds harder to read.
- −Scaling makes features comparable in units, not in importance: it weights every feature equally, which may drown a crucial feature among noisy ones.
- −One-hot explodes with high cardinality and creates rare, noisy columns; hashing and target encoding trade interpretability for size.
- −Target encoding needs cross-fitting and smoothing; done naively it leaks the label and produces optimistic validation scores.
Check yourself
Which of these models gives exactly the same predictions whether or not you standardize the numeric features?
Check yourself
House F's area grows from 240 to 600 m² while A–E stay the same. Which scaler leaves the scaled areas of A–E unchanged?
Check yourself
Nice has one house, priced 520 k€; the global mean price is 335 k€. What is Nice's target encoding with smoothing ?
Check yourself
You one-hot encode a 4-category feature into 4 columns and fit ordinary least squares with an intercept. What goes wrong?
Remember
- ✓Models that use distances, variances, dot products, gradients or norm penalties are sensitive to units; trees are not.
- ✓Standardize by default; min-max for bounded inputs without outliers; robust (median/IQR) when outliers would squash everyone else; max-abs for sparse data; log/Box-Cox/Yeo-Johnson to fix skew.
- ✓Unscaled features make ill-conditioned: elongated contours, zig-zagging gradient descent, iterations ∝ κ. Six houses: κ 131,766 → 15.7 and 668,829 → 59 iterations.
- ✓Scaling changes who your nearest neighbours are, so the scaler is a modelling choice: in the example Q's neighbour moved from C to A (or B).
- ✓One-hot for few nominal categories (drop one only for unregularized linear models); ordinal only for truly ordered ones; target encoding with smoothing and cross-fitting for high cardinality; hashing for open vocabularies.
- ✓Fit every scaler and encoder on training data only, inside a Pipeline; plan for unseen categories.