Definition
Logistic regression is a linear classifier that outputs a probability. It computes a weighted sum of the features, , squashes it into with the sigmoid, , and learns and by minimizing the cross-entropy (log-loss) between those probabilities and the observed labels.
At a glance
- Task
- Binary classification; multi-class via softmax (multinomial) or one-vs-rest
- Input → output
- → probability ; label , usually
- Learns
- a weight per feature and an intercept — the log-odds are
- Training
- convex optimization, no closed form: gradient descent per step, or Newton / IRLS per step
- Prediction
- — one dot product and one sigmoid
- Hyperparameters
- regularization strength , penalty (L2 / L1 / elastic net), class weights, decision threshold
- Assumes
- log-odds linear in the features; independent rows; classes not perfectly separable (or regularize)
The problem
Will this student pass? Is this transaction fraud? Is this tumour malignant? A yes/no label is rarely enough: we want a probability, so that a doctor can act at 10% risk while a spam filter waits for 99%, and so that scores can be ranked, averaged and compared.
The obvious first attempt is to code the labels as 0/1 and fit ordinary linear regression (the linear probability model). On the 10 students of the worked example below, least squares gives : it predicts for a student who studied 0 hours and for one who studied 6 — "probabilities" outside . Worse, squared error punishes a confident correct prediction far from the line, so a few extreme points tilt the fit and drag the 0.5 crossing away from where the classes actually meet.
The perceptron (see its own article) fixes the outlier problem by caring only about the sign of , but it outputs no probability at all and never converges when the classes overlap. Logistic regression keeps the linear score and the hyperplane boundary, and adds exactly what both are missing: a calibrated probability and a smooth, convex loss.
The idea
Probabilities live in ; a linear score lives on the whole real line. So don't model linearly — model a transformation of that also lives on the whole real line. The odds map to ; taking the log maps that to . Logistic regression says: the log-odds are linear in the features. Inverting the log-odds gives the sigmoid.
Geometrically, is a signed distance from a hyperplane (scaled by ). On the hyperplane and ; moving to the positive side, rises smoothly to 1; to the negative side it falls to 0. The size of sets how sharply the probability switches — a large means a near-step, a small one a gentle ramp.
Vocabulary
- Odds
- . A probability of 0.8 is odds of 4 (to 1).
- Log-odds / logit
- , the inverse of the sigmoid. Logistic regression models it as .
- Linear score $z$
- , often itself called "the logit" in deep learning.
- Sigmoid (logistic function)
- ; S-shaped, , .
- Decision boundary
- the set where — a point in 1-D, a line in 2-D, a hyperplane in general.
- Odds ratio
- : how much the odds multiply when feature rises by one unit, others fixed.
- Likelihood
- the probability the model assigns to the labels actually observed, .
- Log-loss / cross-entropy
- the negative log-likelihood per row; the training objective.
- IRLS
- iteratively reweighted least squares — Newton's method for this loss, written as a sequence of weighted linear regressions.
- Separable data
- a hyperplane splits the classes perfectly; then the unregularized optimum does not exist (weights grow forever).
The math
The model. One equation, written two ways:
- feature vector of row ( numbers)
- label of row i, 0 or 1
- weights and intercept — the parameters
- linear score = log-odds of class 1
- predicted probability of class 1
Interpreting a weight. Since , raising by one unit adds to the log-odds, i.e. multiplies the odds by . A weight of 0 means odds ratio 1 (no effect); negative weights shrink the odds. Note it is the odds that change by a constant factor, not the probability: the same moves a lot near 0.5 and hardly at all near 0 or 1.
From likelihood to loss. Treat each label as a coin flip with its own bias: (this is when and when ). Assuming independent rows, the likelihood is the product; maximizing it is the same as minimizing the average negative log:
- log-loss = binary cross-entropy = mean negative log-likelihood
- number of training rows
- cost of a positive row: 0 if p = 1, → ∞ as p → 0
Each row pays of the probability the model gave to its own label: for a confident correct call, for a shrug, for a confident mistake. The loss is unbounded, so confident mistakes dominate — exactly the opposite of squared error's weakness.
Deriving the gradient. Two facts do all the work. First, the sigmoid's derivative is . Second, the per-row loss as a function of is , so by the chain rule
The from the sigmoid cancels the and from the log — this cancellation is why sigmoid and cross-entropy are always paired. Since and :
- the error of row i: positive if it over-predicts class 1
- the matrix of features
It is the same form as linear regression's gradient, with the residual replaced by the probability error. Setting it to zero gives the score equations and — non-linear in because is, so there is no closed form.
Convexity and Newton's method. Differentiating once more gives the Hessian
- with a 1 appended (for the intercept);
- diagonal matrix of weights
Every , so : the loss is convex, any local minimum is global, and gradient descent cannot get stuck. Newton's step uses this curvature; rearranged, each step solves a weighted least-squares problem with weights — hence the name IRLS. Rows near the boundary (, ) get the most weight; rows the model is already sure about get almost none.
Regularization. Add a penalty on the weights (never on ):
- regularization strength; larger λ → smaller weights
- scikit-learn's parameter: , so larger C = weaker penalty
L2 adds to the gradient and to the Hessian: it pulls every weight toward 0 and makes the problem strictly convex, so a unique finite optimum exists even for separable data. Shrinking flattens the sigmoid — predictions move toward 0.5 — and reduces variance. L1 has a kink at 0, so it sets weak features' weights exactly to zero (feature selection). Elastic net mixes the two.
- number of classes; one weight vector per class
- parameters of class k (one class can be fixed to 0)
More than two classes. The softmax above generalizes the sigmoid (with it reduces to it, using ). The loss becomes and the gradient for class is again .
Worked example: hours studied → pass
Ten students, one feature: hours studied , and whether they passed (). The classes overlap — the student who studied 2 hours passed, the ones at 2.5 and 3.5 hours failed — so no threshold is perfect and the probabilities have to do real work.
| Student | Hours x | Passed y | Fitted p | −log p(yᵢ) |
|---|---|---|---|---|
| 1 | 0.5 | 0 | 0.045 | 0.046 |
| 2 | 1.0 | 0 | 0.086 | 0.089 |
| 3 | 1.5 | 0 | 0.156 | 0.169 |
| 4 | 2.0 | 1 | 0.266 | 1.324 |
| 5 | 2.5 | 0 | 0.416 | 0.538 |
| 6 | 3.0 | 1 | 0.584 | 0.538 |
| 7 | 3.5 | 0 | 0.734 | 1.324 |
| 8 | 4.0 | 1 | 0.844 | 0.169 |
| 9 | 4.5 | 1 | 0.914 | 0.089 |
| 10 | 5.0 | 1 | 0.955 | 0.046 |
Fitting it by hand
- 1Start at . Every , so every and the loss is .
- 2Errors. for the five failures and for the five passes. The -gradient is their mean, (the classes are balanced). The -gradient is .
- 3One gradient step with : , stays 0. Now runs from 0.530 (0.5 h) to 0.766 (5 h) and the loss drops to . Gradient descent needs 356 such steps to get within of the minimum.
- 4Newton instead. At the start every , so the Hessian is with determinant . Solving with gives , so and the loss falls to in one step.
- 5Iterate (table below). After 5 Newton steps the gradient is : , , log-loss . Check the score equations: and .
- 6Read the model. Boundary: hours. Odds ratio: — each extra hour multiplies the odds of passing by about 3.9. Training accuracy at : 8/10 (students 4 and 7 are on the wrong side).
- 7Predict a new student who studied 3.8 hours: , — about an 81% chance of passing, odds to 1. At 1.2 hours: , .
| Iteration | w | b | Log-loss | max |gradient| before step |
|---|---|---|---|---|
| 0 | 0 | 0 | 0.6931 | — |
| 1 | 0.9212 | −2.5333 | 0.4514 | 0.475 |
| 2 | 1.2567 | −3.4558 | 0.4343 | 0.091 |
| 3 | 1.3477 | −3.7061 | 0.4335 | 0.016 |
| 4 | 1.3534 | −3.7218 | 0.4335 | 0.0009 |
| 5 | 1.3534 | −3.7219 | 0.4335 | 3×10⁻⁶ |
Seeing it in two dimensions
With two features the boundary is a line and the probabilities form a ramp across the plane. The player below runs plain gradient descent on 40 points: watch the background (predicted probability) sharpen and the contours pull in as grows.
- 1w ← 0, b ← 0
- 2repeat:
- 3 pᵢ ← σ(w·xᵢ + b)
- 4 gᵢ ← pᵢ − yᵢ
- 5 w ← w − η·(1/n)Σ gᵢxᵢ
- 6 b ← b − η·(1/n)Σ gᵢ
- 7until log-loss stops falling
Start with w = (0, 0), b = 0: every point gets p = 0.5, so log-loss = ln 2 = 0.693.
The algorithm, step by step
Fit(X, y)
- 1Prepare. Standardize numeric features (mean 0, variance 1) and one-hot encode categoricals; append a constant column for the intercept or keep
bseparate. - 2Initialize (every ; loss ). The loss is convex, so the start point affects only speed.
- 3Forward. , . Compute stably: for use , and compute the loss from directly (e.g. ).
- 4Gradient. , .
- 5Update. Gradient descent: , . Newton / IRLS: solve and set . Quasi-Newton (L-BFGS) approximates the Hessian from past gradients.
- 6Stop when the gradient norm (or the change in loss) falls below a tolerance
tol, or atmax_iter. - 7Predict. ; label 1 if , with chosen from the costs of false positives and false negatives.
Complexity
- Gradient step
- time, extra memory; SGD / SAGA: per row
- Newton / IRLS step
- to form the Hessian + to solve; few steps (5 on the example)
- L-BFGS step
- with stored gradient pairs — scikit-learn's default
- Prediction
- per row ( for classes)
Variants, hyperparameters and tuning
| C | λ | w | b | Training log-loss | e^w |
|---|---|---|---|---|---|
| 0.1 | 1 | 0.3166 | −0.8705 | 0.5682 | 1.37 |
| 1 (default) | 0.1 | 0.9176 | −2.5233 | 0.4517 | 2.50 |
| 10 | 0.01 | 1.2765 | −3.5103 | 0.4340 | 3.58 |
| ∞ (penalty=None) | 0 | 1.3534 | −3.7219 | 0.4335 | 3.87 |
What to tune
- C (or λ)
- the main knob. Search on a log grid, e.g. , by cross-validated log-loss or AUC. Note sklearn's default is already regularized — on the example it shrinks from 1.35 to 0.92.
- penalty
l2(default; smooth, keeps all features),l1(sparse — zeros out weak features),elasticnet(both; use with correlated groups of features).- class_weight
balancedreweights rows by inverse class frequency, which shifts the intercept toward the rare class. Alternative: keep the weights and move the threshold.- threshold τ
- not a training parameter: pick it afterwards from a precision–recall or cost curve. If a false negative costs and a false positive , the Bayes-optimal threshold for calibrated is .
- solver
lbfgs(default, L2),liblinear(small data, L1),newton-cg/newton-cholesky(exact, few features),saga(large data, L1 / elastic net).- multi-class
- multinomial (softmax, one joint model — the default) or one-vs-rest (K independent binary models).
- features
- the model is linear in what you give it: add interactions, splines or polynomial terms to bend the boundary, then regularize harder.
In code
1import numpy as np23def sigmoid(z):4 return 1 / (1 + np.exp(-z))56def log_loss(X, y, w, b, lam=0.0):7 p = sigmoid(X @ w + b)8 return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)) + lam / 2 * w @ w910def fit_gd(X, y, lr=0.5, lam=0.0, tol=1e-8, max_iter=100_000):11 n, d = X.shape12 w, b = np.zeros(d), 0.013 for it in range(max_iter):14 e = sigmoid(X @ w + b) - y # p_i - y_i15 gw, gb = X.T @ e / n + lam * w, e.mean()16 w, b = w - lr * gw, b - lr * gb17 if max(np.abs(gw).max(), abs(gb)) < tol:18 break19 return w, b, it + 12021def fit_newton(X, y, lam=0.0, tol=1e-10, max_iter=50):22 n, d = X.shape23 A = np.c_[X, np.ones(n)] # last column = intercept24 theta = np.zeros(d + 1)25 R = lam * np.eye(d + 1); R[-1, -1] = 0 # never penalize b26 for it in range(max_iter):27 p = sigmoid(A @ theta)28 g = A.T @ (p - y) / n + R @ theta29 Hs = A.T @ (A * (p * (1 - p))[:, None]) / n + R30 theta -= np.linalg.solve(Hs, g) # Newton / IRLS step31 if np.abs(g).max() < tol:32 break33 return theta[:-1], theta[-1], it + 13435hours = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])36passed = np.array([0, 0, 0, 1, 0, 1, 0, 1, 1, 1])37X = hours[:, None]3839w, b, k = fit_newton(X, passed)40print(f"Newton: w={w[0]:.4f} b={b:.4f} in {k} iterations, loss={log_loss(X, passed, w, b):.4f}")41w2, b2, k2 = fit_gd(X, passed, lr=0.5, tol=1e-6)42print(f"GD: w={w2[0]:.4f} b={b2:.4f} in {k2} iterations")43print(f"boundary x* = {-b / w[0]:.2f} h, odds ratio per hour = {np.exp(w[0]):.3f}")44print(f"P(pass | 3.8 h) = {sigmoid(3.8 * w[0] + b):.3f}")45wr, br, _ = fit_newton(X, passed, lam=0.1)46print(f"L2 lam=0.1: w={wr[0]:.4f} b={br:.4f}") # 0.9176, -2.52331import numpy as np2from sklearn.linear_model import LogisticRegression3from sklearn.metrics import log_loss, accuracy_score45hours = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])6passed = np.array([0, 0, 0, 1, 0, 1, 0, 1, 1, 1])7X = hours[:, None]89clf = LogisticRegression(penalty=None, solver="newton-cg", tol=1e-10).fit(X, passed)10p = clf.predict_proba(X)[:, 1]11print("w =", clf.coef_[0, 0].round(4), " b =", clf.intercept_[0].round(4))12print("train log-loss =", round(log_loss(passed, p), 4), " accuracy =", accuracy_score(passed, clf.predict(X)))13print("P(pass | 1.2 h, 3.8 h) =", clf.predict_proba([[1.2], [3.8]])[:, 1].round(3))1415for C in [0.1, 1, 10]: # C = 1 / (n * lambda)16 m = LogisticRegression(C=C, solver="newton-cg", tol=1e-10).fit(X, passed)17 print(f"C={C:<4} w={m.coef_[0, 0]:.4f} b={m.intercept_[0]:.4f} "18 f"loss={log_loss(passed, m.predict_proba(X)[:, 1]):.4f}")Use when
- +You need probabilities — risk scores, ranking, expected-cost decisions — and a well-calibrated baseline.
- +The model must be explained or audited: is an odds ratio per unit of feature .
- +Data is large and sparse (text, one-hot categories, click logs): training is convex, fast and parallel.
- +You want a strong, hard-to-overfit baseline before reaching for boosting or neural networks.
Avoid when
- −The boundary is strongly curved or driven by interactions you can't engineer — trees, boosting or kernel SVMs fit it directly.
- −Features are many, highly correlated and unregularized: coefficients flip sign and stop being interpretable.
- −The classes are perfectly separable and you need coefficients or p-values without a penalty — the MLE does not exist.
- −Raw images, audio or text where the useful features must be learned — use a network (whose last layer is still logistic regression).
Where it's used
Clinical risk scores
Probability of disease or readmission from age, vitals and lab values; odds ratios are the language of epidemiology.
Credit scoring
Probability of default, often on binned (weight-of-evidence) features; regulators favour its transparency.
Ad click-through rate
Web-scale CTR prediction on billions of sparse hashed features, trained by SGD with L1/L2.
Neural network heads
The sigmoid / softmax output layer trained with cross-entropy is logistic regression on learned features.
Check yourself
In the worked example per hour. Studying one more hour multiplies the odds of passing by about…
Check yourself
A model has and on one feature. Where is the decision boundary at threshold 0.5?
Check yourself
You fit an unregularized logistic regression to data that a straight line separates perfectly. What happens?
Check yourself
After fitting the worked example (with an intercept), what is over the 10 training students?
Remember
- ✓: the log-odds are linear; the boundary is the hyperplane ; is the odds ratio.
- ✓Train by minimizing cross-entropy (the negative log-likelihood). Its gradient is — sigmoid and log cancel.
- ✓The Hessian is positive semi-definite, so the loss is convex; Newton / IRLS converges in a handful of steps, gradient descent in hundreds.
- ✓Regularize (; L2 shrinks, L1 sparsifies) — essential when classes are separable or features many and correlated. Scale features first.
- ✓Use probabilities, not labels: evaluate with log-loss / AUC and set the threshold from error costs.