Logistic Regression

A linear score passed through a sigmoid gives P(y=1|x); it is trained with cross-entropy loss.

17 min read1750 wordscore10 viewsupdated 2026-10-07

Definition

Logistic regression is a linear classifier that outputs a probability. It computes a weighted sum of the features, z=w⊤x+bz=w^\top x+b, squashes it into (0,1)(0,1) with the sigmoid, P(y=1∣x)=σ(z)=1/(1+e−z)P(y=1\mid x)=\sigma(z)=1/(1+e^{-z}), and learns ww and bb by minimizing the cross-entropy (log-loss) between those probabilities and the observed labels.

At a glance

Task
Binary classification; multi-class via softmax (multinomial) or one-vs-rest
Input → output
x∈Rdx\in\mathbb{R}^d → probability p=P(y=1∣x)∈(0,1)p=P(y=1\mid x)\in(0,1); label y^=[ p≥τ ]\hat y=[\,p\ge\tau\,], usually τ=0.5\tau=0.5
Learns
a weight wjw_j per feature and an intercept bb — the log-odds are w⊤x+bw^\top x+b
Training
convex optimization, no closed form: gradient descent O(nd)O(nd) per step, or Newton / IRLS O(nd2+d3)O(nd^2+d^3) per step
Prediction
O(d)O(d) — one dot product and one sigmoid
Hyperparameters
regularization strength C=1/(nλ)C=1/(n\lambda), penalty (L2 / L1 / elastic net), class weights, decision threshold τ\tau
Assumes
log-odds linear in the features; independent rows; classes not perfectly separable (or regularize)

The problem

Will this student pass? Is this transaction fraud? Is this tumour malignant? A yes/no label is rarely enough: we want a probability, so that a doctor can act at 10% risk while a spam filter waits for 99%, and so that scores can be ranked, averaged and compared.

The obvious first attempt is to code the labels as 0/1 and fit ordinary linear regression (the linear probability model). On the 10 students of the worked example below, least squares gives y^=0.230 x−0.133\hat y=0.230\,x-0.133: it predicts −0.13-0.13 for a student who studied 0 hours and 1.251.25 for one who studied 6 — "probabilities" outside [0,1][0,1]. Worse, squared error punishes a confident correct prediction far from the line, so a few extreme points tilt the fit and drag the 0.5 crossing away from where the classes actually meet.

The perceptron (see its own article) fixes the outlier problem by caring only about the sign of w⊤x+bw^\top x+b, but it outputs no probability at all and never converges when the classes overlap. Logistic regression keeps the linear score and the hyperplane boundary, and adds exactly what both are missing: a calibrated probability and a smooth, convex loss.

The idea

Probabilities live in (0,1)(0,1); a linear score lives on the whole real line. So don't model pp linearly — model a transformation of pp that also lives on the whole real line. The odds p/(1−p)p/(1-p) map (0,1)(0,1) to (0,∞)(0,\infty); taking the log maps that to (−∞,∞)(-\infty,\infty). Logistic regression says: the log-odds are linear in the features. Inverting the log-odds gives the sigmoid.

Geometrically, z=w⊤x+bz=w^\top x+b is a signed distance from a hyperplane (scaled by ∥w∥\lVert w\rVert). On the hyperplane z=0z=0 and p=0.5p=0.5; moving to the positive side, pp rises smoothly to 1; to the negative side it falls to 0. The size of ∥w∥\lVert w\rVert sets how sharply the probability switches — a large ∥w∥\lVert w\rVert means a near-step, a small one a gentle ramp.

Vocabulary

Odds
p/(1−p)p/(1-p). A probability of 0.8 is odds of 4 (to 1).
Log-odds / logit
logit⁡(p)=log⁡p1−p\operatorname{logit}(p)=\log\frac{p}{1-p}, the inverse of the sigmoid. Logistic regression models it as w⊤x+bw^\top x+b.
Linear score $z$
z=w⊤x+bz=w^\top x+b, often itself called "the logit" in deep learning.
Sigmoid (logistic function)
σ(z)=1/(1+e−z)\sigma(z)=1/(1+e^{-z}); S-shaped, σ(0)=0.5\sigma(0)=0.5, σ(−z)=1−σ(z)\sigma(-z)=1-\sigma(z).
Decision boundary
the set {x:w⊤x+b=0}\{x: w^\top x+b=0\} where p=0.5p=0.5 — a point in 1-D, a line in 2-D, a hyperplane in general.
Odds ratio
ewje^{w_j}: how much the odds multiply when feature jj rises by one unit, others fixed.
Likelihood
the probability the model assigns to the labels actually observed, ∏ipiyi(1−pi)1−yi\prod_i p_i^{y_i}(1-p_i)^{1-y_i}.
Log-loss / cross-entropy
the negative log-likelihood per row; the training objective.
IRLS
iteratively reweighted least squares — Newton's method for this loss, written as a sequence of weighted linear regressions.
Separable data
a hyperplane splits the classes perfectly; then the unregularized optimum does not exist (weights grow forever).

The math

The model. One equation, written two ways:

pi=P(yi=1∣xi)=σ(zi)=11+e−zi,zi=w⊤xi+b=log⁡pi1−pip_i=P(y_i=1\mid x_i)=\sigma(z_i)=\frac{1}{1+e^{-z_i}},\qquad z_i=w^\top x_i+b=\log\frac{p_i}{1-p_i}
xix_i
feature vector of row ii (dd numbers)
yiy_i
label of row i, 0 or 1
w, bw,\ b
weights and intercept — the parameters
ziz_i
linear score = log-odds of class 1
pip_i
predicted probability of class 1

Interpreting a weight. Since log⁡odds=w⊤x+b\log\text{odds}=w^\top x+b, raising xjx_j by one unit adds wjw_j to the log-odds, i.e. multiplies the odds by ewje^{w_j}. A weight of 0 means odds ratio 1 (no effect); negative weights shrink the odds. Note it is the odds that change by a constant factor, not the probability: the same +wj+w_j moves pp a lot near 0.5 and hardly at all near 0 or 1.

From likelihood to loss. Treat each label as a coin flip with its own bias: P(yi∣xi)=pi yi(1−pi)1−yiP(y_i\mid x_i)=p_i^{\,y_i}(1-p_i)^{1-y_i} (this is pip_i when yi=1y_i=1 and 1−pi1-p_i when yi=0y_i=0). Assuming independent rows, the likelihood is the product; maximizing it is the same as minimizing the average negative log:

L(w,b)=−1n∑i=1n[yilog⁡pi+(1−yi)log⁡(1−pi)]L(w,b)=-\frac{1}{n}\sum_{i=1}^n\Big[y_i\log p_i+(1-y_i)\log(1-p_i)\Big]
LL
log-loss = binary cross-entropy = mean negative log-likelihood
nn
number of training rows
−log⁡pi-\log p_i
cost of a positive row: 0 if p = 1, → ∞ as p → 0

Each row pays −log⁡-\log of the probability the model gave to its own label: −log⁡0.9=0.105-\log 0.9=0.105 for a confident correct call, −log⁡0.5=0.693-\log 0.5=0.693 for a shrug, −log⁡0.1=2.303-\log 0.1=2.303 for a confident mistake. The loss is unbounded, so confident mistakes dominate — exactly the opposite of squared error's weakness.

Deriving the gradient. Two facts do all the work. First, the sigmoid's derivative is σ′(z)=σ(z) (1−σ(z))\sigma'(z)=\sigma(z)\,(1-\sigma(z)). Second, the per-row loss as a function of ziz_i is ℓi=−yilog⁡σ(zi)−(1−yi)log⁡(1−σ(zi))\ell_i=-y_i\log\sigma(z_i)-(1-y_i)\log(1-\sigma(z_i)), so by the chain rule

∂ℓi∂zi=−yipi(1−pi)pi+(1−yi)pi(1−pi)1−pi=−yi(1−pi)+(1−yi)pi=pi−yi\frac{\partial \ell_i}{\partial z_i}=-y_i\frac{p_i(1-p_i)}{p_i}+(1-y_i)\frac{p_i(1-p_i)}{1-p_i}=-y_i(1-p_i)+(1-y_i)p_i=p_i-y_i

The p(1−p)p(1-p) from the sigmoid cancels the 1/p1/p and 1/(1−p)1/(1-p) from the log — this cancellation is why sigmoid and cross-entropy are always paired. Since ∂zi/∂w=xi\partial z_i/\partial w=x_i and ∂zi/∂b=1\partial z_i/\partial b=1:

∇wL=1n∑i=1n(pi−yi) xi=1nX⊤(p−y),∂L∂b=1n∑i=1n(pi−yi)\nabla_w L=\frac{1}{n}\sum_{i=1}^n (p_i-y_i)\,x_i=\tfrac1n X^\top(p-y),\qquad \frac{\partial L}{\partial b}=\frac{1}{n}\sum_{i=1}^n (p_i-y_i)
pi−yip_i-y_i
the error of row i: positive if it over-predicts class 1
XX
the n×dn\times d matrix of features

It is the same form as linear regression's gradient, with the residual replaced by the probability error. Setting it to zero gives the score equations ∑ipi=∑iyi\sum_i p_i=\sum_i y_i and ∑ipixi=∑iyixi\sum_i p_i x_i=\sum_i y_i x_i — non-linear in ww because pip_i is, so there is no closed form.

Convexity and Newton's method. Differentiating once more gives the Hessian

∇2L=1n∑i=1npi(1−pi) x~ix~i⊤=1nX~⊤SX~,θ←θ−(∇2L)−1∇L\nabla^2 L=\frac{1}{n}\sum_{i=1}^n p_i(1-p_i)\,\tilde x_i\tilde x_i^\top=\tfrac1n\tilde X^\top S\tilde X,\qquad \theta\leftarrow\theta-\big(\nabla^2L\big)^{-1}\nabla L
x~i\tilde x_i
xix_i with a 1 appended (for the intercept); θ=(w,b)\theta=(w,b)
SS
diagonal matrix of weights si=pi(1−pi)∈(0,14]s_i=p_i(1-p_i)\in(0,\tfrac14]

Every si>0s_i>0, so v⊤∇2L v=1n∑isi(x~i⊤v)2≥0v^\top\nabla^2L\,v=\tfrac1n\sum_i s_i(\tilde x_i^\top v)^2\ge0: the loss is convex, any local minimum is global, and gradient descent cannot get stuck. Newton's step uses this curvature; rearranged, each step solves a weighted least-squares problem with weights sis_i — hence the name IRLS. Rows near the boundary (p≈0.5p\approx0.5, si=0.25s_i=0.25) get the most weight; rows the model is already sure about get almost none.

Regularization. Add a penalty on the weights (never on bb):

Lλ(w,b)=L(w,b)+λ2∥w∥22(L2)orL(w,b)+λ∥w∥1(L1)L_\lambda(w,b)=L(w,b)+\frac{\lambda}{2}\lVert w\rVert_2^2\quad\text{(L2)}\qquad\text{or}\qquad L(w,b)+\lambda\lVert w\rVert_1\quad\text{(L1)}
λ\lambda
regularization strength; larger λ → smaller weights
CC
scikit-learn's parameter: C=1/(nλ)C=1/(n\lambda), so larger C = weaker penalty

L2 adds λw\lambda w to the gradient and λI\lambda I to the Hessian: it pulls every weight toward 0 and makes the problem strictly convex, so a unique finite optimum exists even for separable data. Shrinking ∥w∥\lVert w\rVert flattens the sigmoid — predictions move toward 0.5 — and reduces variance. L1 has a kink at 0, so it sets weak features' weights exactly to zero (feature selection). Elastic net mixes the two.

P(y=k∣x)=ewk⊤x+bk∑j=1Kewj⊤x+bjP(y=k\mid x)=\frac{e^{w_k^\top x+b_k}}{\sum_{j=1}^K e^{w_j^\top x+b_j}}
KK
number of classes; one weight vector per class
wk, bkw_k,\ b_k
parameters of class k (one class can be fixed to 0)

More than two classes. The softmax above generalizes the sigmoid (with K=2K=2 it reduces to it, using w=w1−w0w=w_1-w_0). The loss becomes −1n∑ilog⁡P(yi∣xi)-\frac1n\sum_i\log P(y_i\mid x_i) and the gradient for class kk is again 1n∑i(pik−[yi=k]) xi\frac1n\sum_i(p_{ik}-[y_i=k])\,x_i.

Worked example: hours studied → pass

Ten students, one feature: hours studied xx, and whether they passed (y=1y=1). The classes overlap — the student who studied 2 hours passed, the ones at 2.5 and 3.5 hours failed — so no threshold is perfect and the probabilities have to do real work.

StudentHours xPassed yFitted p−log p(yᵢ)
10.500.0450.046
21.000.0860.089
31.500.1560.169
42.010.2661.324
52.500.4160.538
63.010.5840.538
73.500.7341.324
84.010.8440.169
94.510.9140.089
105.010.9550.046
Training data, with the fitted probabilities and each row's log-loss at the optimum (w = 1.3534, b = −3.7219). Highlighted: the two students the 0.5 threshold misclassifies.

Fitting it by hand

  1. 1Start at w=b=0w=b=0. Every zi=0z_i=0, so every pi=0.5p_i=0.5 and the loss is −log⁡0.5=ln⁡2=0.6931-\log 0.5=\ln 2=0.6931.
  2. 2Errors. pi−yi=+0.5p_i-y_i=+0.5 for the five failures and −0.5-0.5 for the five passes. The bb-gradient is their mean, 00 (the classes are balanced). The ww-gradient is 110∑(pi−yi)xi=110[0.5(0.5+1+1.5+2.5+3.5)−0.5(2+3+4+4.5+5)]=110(4.5−9.25)=−0.475\tfrac1{10}\sum(p_i-y_i)x_i=\tfrac1{10}\big[0.5(0.5+1+1.5+2.5+3.5)-0.5(2+3+4+4.5+5)\big]=\tfrac1{10}(4.5-9.25)=-0.475.
  3. 3One gradient step with η=0.5\eta=0.5: w←0−0.5(−0.475)=0.2375w\leftarrow0-0.5(-0.475)=0.2375, bb stays 0. Now pp runs from 0.530 (0.5 h) to 0.766 (5 h) and the loss drops to 0.64580.6458. Gradient descent needs 356 such steps to get within 10−410^{-4} of the minimum.
  4. 4Newton instead. At the start every si=0.25s_i=0.25, so the Hessian is (0.25x2‾0.25x‾0.25x‾0.25)=(2.406250.68750.68750.25)\begin{pmatrix}\overline{0.25x^2}&\overline{0.25x}\\\overline{0.25x}&0.25\end{pmatrix}=\begin{pmatrix}2.40625&0.6875\\0.6875&0.25\end{pmatrix} with determinant 0.128910.12891. Solving HΔ=gH\Delta=g with g=(−0.475,0)g=(-0.475,0) gives Δ=(−0.9212, 2.5333)\Delta=(-0.9212,\,2.5333), so (w,b)←(0.9212, −2.5333)(w,b)\leftarrow(0.9212,\,-2.5333) and the loss falls to 0.45140.4514 in one step.
  5. 5Iterate (table below). After 5 Newton steps the gradient is 3×10−63\times10^{-6}: w=1.3534w=1.3534, b=−3.7219b=-3.7219, log-loss 0.43350.4335. Check the score equations: ∑pi=5.000=∑yi\sum p_i=5.000=\sum y_i and ∑pixi=18.50=∑yixi\sum p_i x_i=18.50=\sum y_ix_i.
  6. 6Read the model. Boundary: z=0  ⟺  x∗=−b/w=3.7219/1.3534=2.75z=0\iff x^*=-b/w=3.7219/1.3534=2.75 hours. Odds ratio: e1.3534=3.87e^{1.3534}=3.87 — each extra hour multiplies the odds of passing by about 3.9. Training accuracy at τ=0.5\tau=0.5: 8/10 (students 4 and 7 are on the wrong side).
  7. 7Predict a new student who studied 3.8 hours: z=1.3534(3.8)−3.7219=1.421z=1.3534(3.8)-3.7219=1.421, p=σ(1.421)=0.806p=\sigma(1.421)=0.806 — about an 81% chance of passing, odds e1.421=4.1e^{1.421}=4.1 to 1. At 1.2 hours: z=−2.098z=-2.098, p=0.109p=0.109.
IterationwbLog-lossmax |gradient| before step
0000.6931—
10.9212−2.53330.45140.475
21.2567−3.45580.43430.091
31.3477−3.70610.43350.016
41.3534−3.72180.43350.0009
51.3534−3.72190.43353×10⁻⁶
Newton / IRLS iterations from w = b = 0. The loss is within 3×10⁻⁶ of its minimum after three steps; gradient descent (η = 0.5) needs 356 steps for 10⁻⁴.
P(pass | hours) = σ(w·x + b) · ● passed ● failed · drag the boundary00.510h1h2h3h4h5hlog-loss over (w, b) · drag the dot0123w →0-4-8bper-point −log p(yᵢ)012312345hours →
w = 0.000 · b = 0.000 · odds ratio ew = 1.00
objective = 0.6931 · minimum = 0.4335 at (w, b) = (1.353, -3.722)
gradient (∂/∂w, ∂/∂b) = (-0.4750, 0.0000) · accuracy 5/10
w = b = 0: every p = 0.5, log-loss = ln 2. Take a step or drag.
λ
fig.The ten students. Top: the sigmoid σ(w·x + b), the boundary x* = −b/w (drag it), and each row's error pᵢ − yᵢ as a rose dash; ringed points are misclassified. Bottom left: the log-loss over (w, b) — a single convex valley, long and narrow — with the minimum marked ✕; drag the orange dot anywhere. Bottom right: each row's loss −log p(yᵢ). From w = b = 0 (loss 0.6931), one GD step (η = 0.5) reaches (0.238, 0); Newton reaches the minimum 0.4335 at (1.353, −3.722) in 4–5 clicks while GD crawls along the valley floor (it needs 7–8 clicks of GD ×50 to come within 0.0001). λ = 0.1 (sklearn C = 1) moves the minimum to (0.918, −2.523) — a flatter sigmoid with the same boundary 2.75 h.

Seeing it in two dimensions

With two features the boundary is a line and the probabilities form a ramp across the plane. The player below runs plain gradient descent on 40 points: watch the background (predicted probability) sharpen and the p=0.1/0.9p=0.1/0.9 contours pull in as ∥w∥\lVert w\rVert grows.

▸ Logistic regression · gradient descentstep 1/20
iteration 0log-loss = 0.6931accuracy —
  1. 1w ← 0, b ← 0
  2. 2repeat:
  3. 3 pᵢ ← σ(w·xᵢ + b)
  4. 4 gᵢ ← pᵢ − yᵢ
  5. 5 w ← w − η·(1/n)Σ gᵢxᵢ
  6. 6 b ← b − η·(1/n)Σ gᵢ
  7. 7until log-loss stops falling

Start with w = (0, 0), b = 0: every point gets p = 0.5, so log-loss = ln 2 = 0.693.

fig.Background = predicted P(class 1); the solid line is p = 0.5, the dashed lines p = 0.1 and 0.9. Rings show each point's error pᵢ − yᵢ. Log-loss starts at ln 2 = 0.693 and the remaining mistakes sit where the classes overlap.

The algorithm, step by step

Fit(X, y)

  1. 1Prepare. Standardize numeric features (mean 0, variance 1) and one-hot encode categoricals; append a constant column for the intercept or keep b separate.
  2. 2Initialize w=0, b=0w=0,\ b=0 (every pi=0.5p_i=0.5; loss =ln⁡2=\ln 2). The loss is convex, so the start point affects only speed.
  3. 3Forward. z=Xw+bz=Xw+b, p=σ(z)p=\sigma(z). Compute σ\sigma stably: for z<0z<0 use ez/(1+ez)e^{z}/(1+e^{z}), and compute the loss from zz directly (e.g. log⁡(1+ez)−yz\log(1+e^{z})-yz).
  4. 4Gradient. gw=1nX⊤(p−y)+λwg_w=\tfrac1nX^\top(p-y)+\lambda w, gb=1n∑(pi−yi)g_b=\tfrac1n\sum(p_i-y_i).
  5. 5Update. Gradient descent: w←w−ηgww\leftarrow w-\eta g_w, b←b−ηgbb\leftarrow b-\eta g_b. Newton / IRLS: solve (1nX~⊤SX~+λI′)Δ=g(\tfrac1n\tilde X^\top S\tilde X+\lambda I')\Delta=g and set θ←θ−Δ\theta\leftarrow\theta-\Delta. Quasi-Newton (L-BFGS) approximates the Hessian from past gradients.
  6. 6Stop when the gradient norm (or the change in loss) falls below a tolerance tol, or at max_iter.
  7. 7Predict. p=σ(w⊤x+b)p=\sigma(w^\top x+b); label 1 if p≥τp\ge\tau, with τ\tau chosen from the costs of false positives and false negatives.

Complexity

Gradient step
O(nd)O(nd) time, O(d)O(d) extra memory; SGD / SAGA: O(d)O(d) per row
Newton / IRLS step
O(nd2)O(nd^2) to form the Hessian + O(d3)O(d^3) to solve; few steps (5 on the example)
L-BFGS step
O(nd+md)O(nd + md) with m≈10m\approx10 stored gradient pairs — scikit-learn's default
Prediction
O(d)O(d) per row (O(Kd)O(Kd) for KK classes)

Variants, hyperparameters and tuning

CλwbTraining log-losse^w
0.110.3166−0.87050.56821.37
1 (default)0.10.9176−2.52330.45172.50
100.011.2765−3.51030.43403.58
∞ (penalty=None)01.3534−3.72190.43353.87
L2 strength on the worked example (scikit-learn, C = 1/(nλ) with n = 10). Stronger penalties shrink w toward 0 and raise the training log-loss; the boundary stays at 2.75 h and accuracy at 8/10.

What to tune

C (or λ)
the main knob. Search on a log grid, e.g. 10−3…10310^{-3}\ldots10^{3}, by cross-validated log-loss or AUC. Note sklearn's default C=1C=1 is already regularized — on the example it shrinks ww from 1.35 to 0.92.
penalty
l2 (default; smooth, keeps all features), l1 (sparse — zeros out weak features), elasticnet (both; use with correlated groups of features).
class_weight
balanced reweights rows by inverse class frequency, which shifts the intercept toward the rare class. Alternative: keep the weights and move the threshold.
threshold τ
not a training parameter: pick it afterwards from a precision–recall or cost curve. If a false negative costs cFNc_{FN} and a false positive cFPc_{FP}, the Bayes-optimal threshold for calibrated pp is τ=cFP/(cFP+cFN)\tau=c_{FP}/(c_{FP}+c_{FN}).
solver
lbfgs (default, L2), liblinear (small data, L1), newton-cg / newton-cholesky (exact, few features), saga (large data, L1 / elastic net).
multi-class
multinomial (softmax, one joint model — the default) or one-vs-rest (K independent binary models).
features
the model is linear in what you give it: add interactions, splines or polynomial terms to bend the boundary, then regularize harder.

In code

python
1import numpy as np
2
3def sigmoid(z):
4 return 1 / (1 + np.exp(-z))
5
6def log_loss(X, y, w, b, lam=0.0):
7 p = sigmoid(X @ w + b)
8 return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)) + lam / 2 * w @ w
9
10def fit_gd(X, y, lr=0.5, lam=0.0, tol=1e-8, max_iter=100_000):
11 n, d = X.shape
12 w, b = np.zeros(d), 0.0
13 for it in range(max_iter):
14 e = sigmoid(X @ w + b) - y # p_i - y_i
15 gw, gb = X.T @ e / n + lam * w, e.mean()
16 w, b = w - lr * gw, b - lr * gb
17 if max(np.abs(gw).max(), abs(gb)) < tol:
18 break
19 return w, b, it + 1
20
21def fit_newton(X, y, lam=0.0, tol=1e-10, max_iter=50):
22 n, d = X.shape
23 A = np.c_[X, np.ones(n)] # last column = intercept
24 theta = np.zeros(d + 1)
25 R = lam * np.eye(d + 1); R[-1, -1] = 0 # never penalize b
26 for it in range(max_iter):
27 p = sigmoid(A @ theta)
28 g = A.T @ (p - y) / n + R @ theta
29 Hs = A.T @ (A * (p * (1 - p))[:, None]) / n + R
30 theta -= np.linalg.solve(Hs, g) # Newton / IRLS step
31 if np.abs(g).max() < tol:
32 break
33 return theta[:-1], theta[-1], it + 1
34
35hours = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
36passed = np.array([0, 0, 0, 1, 0, 1, 0, 1, 1, 1])
37X = hours[:, None]
38
39w, b, k = fit_newton(X, passed)
40print(f"Newton: w={w[0]:.4f} b={b:.4f} in {k} iterations, loss={log_loss(X, passed, w, b):.4f}")
41w2, b2, k2 = fit_gd(X, passed, lr=0.5, tol=1e-6)
42print(f"GD: w={w2[0]:.4f} b={b2:.4f} in {k2} iterations")
43print(f"boundary x* = {-b / w[0]:.2f} h, odds ratio per hour = {np.exp(w[0]):.3f}")
44print(f"P(pass | 3.8 h) = {sigmoid(3.8 * w[0] + b):.3f}")
45wr, br, _ = fit_newton(X, passed, lam=0.1)
46print(f"L2 lam=0.1: w={wr[0]:.4f} b={br:.4f}") # 0.9176, -2.5233
From scratch: gradient descent and Newton / IRLS, with optional L2. Prints w = 1.3534, b = −3.7219 (6 Newton iterations, the last only confirming the gradient is 0; 1,255 GD iterations to gradient < 10⁻⁶), boundary 2.75 h, odds ratio 3.871, P(pass | 3.8 h) = 0.806.
python
1import numpy as np
2from sklearn.linear_model import LogisticRegression
3from sklearn.metrics import log_loss, accuracy_score
4
5hours = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
6passed = np.array([0, 0, 0, 1, 0, 1, 0, 1, 1, 1])
7X = hours[:, None]
8
9clf = LogisticRegression(penalty=None, solver="newton-cg", tol=1e-10).fit(X, passed)
10p = clf.predict_proba(X)[:, 1]
11print("w =", clf.coef_[0, 0].round(4), " b =", clf.intercept_[0].round(4))
12print("train log-loss =", round(log_loss(passed, p), 4), " accuracy =", accuracy_score(passed, clf.predict(X)))
13print("P(pass | 1.2 h, 3.8 h) =", clf.predict_proba([[1.2], [3.8]])[:, 1].round(3))
14
15for C in [0.1, 1, 10]: # C = 1 / (n * lambda)
16 m = LogisticRegression(C=C, solver="newton-cg", tol=1e-10).fit(X, passed)
17 print(f"C={C:<4} w={m.coef_[0, 0]:.4f} b={m.intercept_[0]:.4f} "
18 f"loss={log_loss(passed, m.predict_proba(X)[:, 1]):.4f}")
scikit-learn on the same data. Prints w = 1.3534, b = −3.7219, log-loss 0.4335, accuracy 0.8, P(pass) = 0.109 and 0.806, then the C table above.

Use when

  • +You need probabilities — risk scores, ranking, expected-cost decisions — and a well-calibrated baseline.
  • +The model must be explained or audited: ewje^{w_j} is an odds ratio per unit of feature jj.
  • +Data is large and sparse (text, one-hot categories, click logs): training is convex, fast and parallel.
  • +You want a strong, hard-to-overfit baseline before reaching for boosting or neural networks.

Avoid when

  • −The boundary is strongly curved or driven by interactions you can't engineer — trees, boosting or kernel SVMs fit it directly.
  • −Features are many, highly correlated and unregularized: coefficients flip sign and stop being interpretable.
  • −The classes are perfectly separable and you need coefficients or p-values without a penalty — the MLE does not exist.
  • −Raw images, audio or text where the useful features must be learned — use a network (whose last layer is still logistic regression).

Where it's used

Clinical risk scores

Probability of disease or readmission from age, vitals and lab values; odds ratios are the language of epidemiology.

Credit scoring

Probability of default, often on binned (weight-of-evidence) features; regulators favour its transparency.

Ad click-through rate

Web-scale CTR prediction on billions of sparse hashed features, trained by SGD with L1/L2.

Neural network heads

The sigmoid / softmax output layer trained with cross-entropy is logistic regression on learned features.

Check yourself

In the worked example w=1.3534w=1.3534 per hour. Studying one more hour multiplies the odds of passing by about…

Check yourself

A model has w=2w=2 and b=−6b=-6 on one feature. Where is the decision boundary at threshold 0.5?

Check yourself

You fit an unregularized logistic regression to data that a straight line separates perfectly. What happens?

Check yourself

After fitting the worked example (with an intercept), what is ∑ipi\sum_i p_i over the 10 training students?

Remember

  • ✓p=σ(w⊤x+b)p=\sigma(w^\top x+b): the log-odds are linear; the boundary is the hyperplane w⊤x+b=0w^\top x+b=0; ewje^{w_j} is the odds ratio.
  • ✓Train by minimizing cross-entropy (the negative log-likelihood). Its gradient is 1nX⊤(p−y)\tfrac1nX^\top(p-y) — sigmoid and log cancel.
  • ✓The Hessian 1nX~⊤SX~\tfrac1n\tilde X^\top S\tilde X is positive semi-definite, so the loss is convex; Newton / IRLS converges in a handful of steps, gradient descent in hundreds.
  • ✓Regularize (C=1/(nλ)C=1/(n\lambda); L2 shrinks, L1 sparsifies) — essential when classes are separable or features many and correlated. Scale features first.
  • ✓Use probabilities, not labels: evaluate with log-loss / AUC and set the threshold from error costs.
← back to ML Algorithms roadmap