# Logistic Regression

> A linear score passed through a sigmoid gives P(y=1|x); it is trained with cross-entropy loss.

Source: https://www.mlfoundry.in/learn/logistic-regression  
Section: ML Algorithms › Supervised · Classification  
Level: core · 17 min read · updated 2026-10-07

**Logistic regression:** is a linear classifier that outputs a **probability**. It computes a weighted sum of the features, $z=w^\top x+b$, squashes it into $(0,1)$ with the sigmoid, $P(y=1\mid x)=\sigma(z)=1/(1+e^{-z})$, and learns $w$ and $b$ by minimizing the cross-entropy (log-loss) between those probabilities and the observed labels.

- **Task:** Binary classification; multi-class via softmax (multinomial) or one-vs-rest
- **Input → output:** $x\in\mathbb{R}^d$ → probability $p=P(y=1\mid x)\in(0,1)$; label $\hat y=[\,p\ge\tau\,]$, usually $\tau=0.5$
- **Learns:** a weight $w_j$ per feature and an intercept $b$ — the log-odds are $w^\top x+b$
- **Training:** convex optimization, no closed form: gradient descent $O(nd)$ per step, or Newton / IRLS $O(nd^2+d^3)$ per step
- **Prediction:** $O(d)$ — one dot product and one sigmoid
- **Hyperparameters:** regularization strength $C=1/(n\lambda)$, penalty (L2 / L1 / elastic net), class weights, decision threshold $\tau$
- **Assumes:** log-odds linear in the features; independent rows; classes not perfectly separable (or regularize)

## The problem

Will this student pass? Is this transaction fraud? Is this tumour malignant? A yes/no label is rarely enough: we want a **probability**, so that a doctor can act at 10% risk while a spam filter waits for 99%, and so that scores can be ranked, averaged and compared.

The obvious first attempt is to code the labels as 0/1 and fit ordinary linear regression (the *linear probability model*). On the 10 students of the worked example below, least squares gives $\hat y=0.230\,x-0.133$: it predicts $-0.13$ for a student who studied 0 hours and $1.25$ for one who studied 6 — "probabilities" outside $[0,1]$. Worse, squared error punishes a confident *correct* prediction far from the line, so a few extreme points tilt the fit and drag the 0.5 crossing away from where the classes actually meet.

The **perceptron** (see its own article) fixes the outlier problem by caring only about the sign of $w^\top x+b$, but it outputs no probability at all and never converges when the classes overlap. Logistic regression keeps the linear score and the hyperplane boundary, and adds exactly what both are missing: a calibrated probability and a smooth, convex loss.

## The idea

Probabilities live in $(0,1)$; a linear score lives on the whole real line. So don't model $p$ linearly — model a transformation of $p$ that *also* lives on the whole real line. The **odds** $p/(1-p)$ map $(0,1)$ to $(0,\infty)$; taking the log maps that to $(-\infty,\infty)$. Logistic regression says: the **log-odds are linear in the features**. Inverting the log-odds gives the sigmoid.

Geometrically, $z=w^\top x+b$ is a signed distance from a hyperplane (scaled by $\lVert w\rVert$). On the hyperplane $z=0$ and $p=0.5$; moving to the positive side, $p$ rises smoothly to 1; to the negative side it falls to 0. The size of $\lVert w\rVert$ sets how *sharply* the probability switches — a large $\lVert w\rVert$ means a near-step, a small one a gentle ramp.

**Vocabulary**

- **Odds:** $p/(1-p)$. A probability of 0.8 is odds of 4 (to 1).
- **Log-odds / logit:** $\operatorname{logit}(p)=\log\frac{p}{1-p}$, the inverse of the sigmoid. Logistic regression models it as $w^\top x+b$.
- **Linear score $z$:** $z=w^\top x+b$, often itself called "the logit" in deep learning.
- **Sigmoid (logistic function):** $\sigma(z)=1/(1+e^{-z})$; S-shaped, $\sigma(0)=0.5$, $\sigma(-z)=1-\sigma(z)$.
- **Decision boundary:** the set $\{x: w^\top x+b=0\}$ where $p=0.5$ — a point in 1-D, a line in 2-D, a hyperplane in general.
- **Odds ratio:** $e^{w_j}$: how much the odds multiply when feature $j$ rises by one unit, others fixed.
- **Likelihood:** the probability the model assigns to the labels actually observed, $\prod_i p_i^{y_i}(1-p_i)^{1-y_i}$.
- **Log-loss / cross-entropy:** the negative log-likelihood per row; the training objective.
- **IRLS:** iteratively reweighted least squares — Newton's method for this loss, written as a sequence of weighted linear regressions.
- **Separable data:** a hyperplane splits the classes perfectly; then the unregularized optimum does not exist (weights grow forever).

## The math

**The model.** One equation, written two ways:

$$
p_i=P(y_i=1\mid x_i)=\sigma(z_i)=\frac{1}{1+e^{-z_i}},\qquad z_i=w^\top x_i+b=\log\frac{p_i}{1-p_i}
$$

- $x_i$: feature vector of row $i$ ($d$ numbers)
- $y_i$: label of row i, 0 or 1
- $w,\ b$: weights and intercept — the parameters
- $z_i$: linear score = log-odds of class 1
- $p_i$: predicted probability of class 1

**Interpreting a weight.** Since $\log\text{odds}=w^\top x+b$, raising $x_j$ by one unit adds $w_j$ to the log-odds, i.e. multiplies the odds by $e^{w_j}$. A weight of 0 means odds ratio 1 (no effect); negative weights shrink the odds. Note it is the *odds* that change by a constant factor, not the probability: the same $+w_j$ moves $p$ a lot near 0.5 and hardly at all near 0 or 1.

**From likelihood to loss.** Treat each label as a coin flip with its own bias: $P(y_i\mid x_i)=p_i^{\,y_i}(1-p_i)^{1-y_i}$ (this is $p_i$ when $y_i=1$ and $1-p_i$ when $y_i=0$). Assuming independent rows, the likelihood is the product; maximizing it is the same as minimizing the average negative log:

$$
L(w,b)=-\frac{1}{n}\sum_{i=1}^n\Big[y_i\log p_i+(1-y_i)\log(1-p_i)\Big]
$$

- $L$: log-loss = binary cross-entropy = mean negative log-likelihood
- $n$: number of training rows
- $-\log p_i$: cost of a positive row: 0 if p = 1, → ∞ as p → 0

Each row pays $-\log$ of the probability the model gave to *its own* label: $-\log 0.9=0.105$ for a confident correct call, $-\log 0.5=0.693$ for a shrug, $-\log 0.1=2.303$ for a confident mistake. The loss is unbounded, so confident mistakes dominate — exactly the opposite of squared error's weakness.

**Deriving the gradient.** Two facts do all the work. First, the sigmoid's derivative is $\sigma'(z)=\sigma(z)\,(1-\sigma(z))$. Second, the per-row loss as a function of $z_i$ is $\ell_i=-y_i\log\sigma(z_i)-(1-y_i)\log(1-\sigma(z_i))$, so by the chain rule

$$
\frac{\partial \ell_i}{\partial z_i}=-y_i\frac{p_i(1-p_i)}{p_i}+(1-y_i)\frac{p_i(1-p_i)}{1-p_i}=-y_i(1-p_i)+(1-y_i)p_i=p_i-y_i
$$

The $p(1-p)$ from the sigmoid cancels the $1/p$ and $1/(1-p)$ from the log — this cancellation is why sigmoid and cross-entropy are always paired. Since $\partial z_i/\partial w=x_i$ and $\partial z_i/\partial b=1$:

$$
\nabla_w L=\frac{1}{n}\sum_{i=1}^n (p_i-y_i)\,x_i=\tfrac1n X^\top(p-y),\qquad \frac{\partial L}{\partial b}=\frac{1}{n}\sum_{i=1}^n (p_i-y_i)
$$

- $p_i-y_i$: the error of row i: positive if it over-predicts class 1
- $X$: the $n\times d$ matrix of features

It is the same form as linear regression's gradient, with the residual replaced by the probability error. Setting it to zero gives the **score equations** $\sum_i p_i=\sum_i y_i$ and $\sum_i p_i x_i=\sum_i y_i x_i$ — non-linear in $w$ because $p_i$ is, so there is no closed form.

**Convexity and Newton's method.** Differentiating once more gives the Hessian

$$
\nabla^2 L=\frac{1}{n}\sum_{i=1}^n p_i(1-p_i)\,\tilde x_i\tilde x_i^\top=\tfrac1n\tilde X^\top S\tilde X,\qquad \theta\leftarrow\theta-\big(\nabla^2L\big)^{-1}\nabla L
$$

- $\tilde x_i$: $x_i$ with a 1 appended (for the intercept); $\theta=(w,b)$
- $S$: diagonal matrix of weights $s_i=p_i(1-p_i)\in(0,\tfrac14]$

Every $s_i>0$, so $v^\top\nabla^2L\,v=\tfrac1n\sum_i s_i(\tilde x_i^\top v)^2\ge0$: the loss is **convex**, any local minimum is global, and gradient descent cannot get stuck. Newton's step uses this curvature; rearranged, each step solves a weighted least-squares problem with weights $s_i$ — hence the name **IRLS**. Rows near the boundary ($p\approx0.5$, $s_i=0.25$) get the most weight; rows the model is already sure about get almost none.

**Regularization.** Add a penalty on the weights (never on $b$):

$$
L_\lambda(w,b)=L(w,b)+\frac{\lambda}{2}\lVert w\rVert_2^2\quad\text{(L2)}\qquad\text{or}\qquad L(w,b)+\lambda\lVert w\rVert_1\quad\text{(L1)}
$$

- $\lambda$: regularization strength; larger λ → smaller weights
- $C$: scikit-learn's parameter: $C=1/(n\lambda)$, so *larger C = weaker penalty*

L2 adds $\lambda w$ to the gradient and $\lambda I$ to the Hessian: it pulls every weight toward 0 and makes the problem *strictly* convex, so a unique finite optimum exists even for separable data. Shrinking $\lVert w\rVert$ flattens the sigmoid — predictions move toward 0.5 — and reduces variance. L1 has a kink at 0, so it sets weak features' weights **exactly** to zero (feature selection). Elastic net mixes the two.

$$
P(y=k\mid x)=\frac{e^{w_k^\top x+b_k}}{\sum_{j=1}^K e^{w_j^\top x+b_j}}
$$

- $K$: number of classes; one weight vector per class
- $w_k,\ b_k$: parameters of class k (one class can be fixed to 0)

**More than two classes.** The softmax above generalizes the sigmoid (with $K=2$ it reduces to it, using $w=w_1-w_0$). The loss becomes $-\frac1n\sum_i\log P(y_i\mid x_i)$ and the gradient for class $k$ is again $\frac1n\sum_i(p_{ik}-[y_i=k])\,x_i$.

> **Discriminative twin of LDA and Naive Bayes:** If each class is Gaussian with a *shared* covariance, Bayes' rule gives a posterior of exactly the logistic form $\sigma(w^\top x+b)$ — that is **linear discriminant analysis**. Gaussian **naive Bayes** with shared variances does too. Those models estimate class densities and *derive* $w$; logistic regression fits $w$ directly to $P(y\mid x)$ and assumes nothing about how $x$ is distributed. It is more robust when the Gaussian story is wrong, while the generative models need less data when it is right. See the LDA, QDA and naive Bayes articles for that side.

## Worked example: hours studied → pass

Ten students, one feature: hours studied $x$, and whether they passed ($y=1$). The classes overlap — the student who studied 2 hours passed, the ones at 2.5 and 3.5 hours failed — so no threshold is perfect and the probabilities have to do real work.

Training data, with the fitted probabilities and each row's log-loss at the optimum (w = 1.3534, b = −3.7219). Highlighted: the two students the 0.5 threshold misclassifies.

| Student | Hours x | Passed y | Fitted p | −log p(yᵢ) |
| --- | --- | --- | --- | --- |
| 1 | 0.5 | 0 | 0.045 | 0.046 |
| 2 | 1.0 | 0 | 0.086 | 0.089 |
| 3 | 1.5 | 0 | 0.156 | 0.169 |
| 4 | 2.0 | 1 | 0.266 | 1.324 |
| 5 | 2.5 | 0 | 0.416 | 0.538 |
| 6 | 3.0 | 1 | 0.584 | 0.538 |
| 7 | 3.5 | 0 | 0.734 | 1.324 |
| 8 | 4.0 | 1 | 0.844 | 0.169 |
| 9 | 4.5 | 1 | 0.914 | 0.089 |
| 10 | 5.0 | 1 | 0.955 | 0.046 |

**Fitting it by hand**

1. **Start** at $w=b=0$. Every $z_i=0$, so every $p_i=0.5$ and the loss is $-\log 0.5=\ln 2=0.6931$.
2. **Errors.** $p_i-y_i=+0.5$ for the five failures and $-0.5$ for the five passes. The $b$-gradient is their mean, $0$ (the classes are balanced). The $w$-gradient is $\tfrac1{10}\sum(p_i-y_i)x_i=\tfrac1{10}\big[0.5(0.5+1+1.5+2.5+3.5)-0.5(2+3+4+4.5+5)\big]=\tfrac1{10}(4.5-9.25)=-0.475$.
3. **One gradient step** with $\eta=0.5$: $w\leftarrow0-0.5(-0.475)=0.2375$, $b$ stays 0. Now $p$ runs from 0.530 (0.5 h) to 0.766 (5 h) and the loss drops to $0.6458$. Gradient descent needs **356 such steps** to get within $10^{-4}$ of the minimum.
4. **Newton instead.** At the start every $s_i=0.25$, so the Hessian is $\begin{pmatrix}\overline{0.25x^2}&\overline{0.25x}\\\overline{0.25x}&0.25\end{pmatrix}=\begin{pmatrix}2.40625&0.6875\\0.6875&0.25\end{pmatrix}$ with determinant $0.12891$. Solving $H\Delta=g$ with $g=(-0.475,0)$ gives $\Delta=(-0.9212,\,2.5333)$, so $(w,b)\leftarrow(0.9212,\,-2.5333)$ and the loss falls to $0.4514$ in one step.
5. **Iterate** (table below). After 5 Newton steps the gradient is $3\times10^{-6}$: $w=1.3534$, $b=-3.7219$, log-loss $0.4335$. Check the score equations: $\sum p_i=5.000=\sum y_i$ and $\sum p_i x_i=18.50=\sum y_ix_i$.
6. **Read the model.** Boundary: $z=0\iff x^*=-b/w=3.7219/1.3534=2.75$ hours. Odds ratio: $e^{1.3534}=3.87$ — each extra hour multiplies the odds of passing by about 3.9. Training accuracy at $\tau=0.5$: 8/10 (students 4 and 7 are on the wrong side).
7. **Predict** a new student who studied 3.8 hours: $z=1.3534(3.8)-3.7219=1.421$, $p=\sigma(1.421)=0.806$ — about an 81% chance of passing, odds $e^{1.421}=4.1$ to 1. At 1.2 hours: $z=-2.098$, $p=0.109$.

Newton / IRLS iterations from w = b = 0. The loss is within 3×10⁻⁶ of its minimum after three steps; gradient descent (η = 0.5) needs 356 steps for 10⁻⁴.

| Iteration | w | b | Log-loss | max \|gradient\| before step |
| --- | --- | --- | --- | --- |
| 0 | 0 | 0 | 0.6931 | — |
| 1 | 0.9212 | −2.5333 | 0.4514 | 0.475 |
| 2 | 1.2567 | −3.4558 | 0.4343 | 0.091 |
| 3 | 1.3477 | −3.7061 | 0.4335 | 0.016 |
| 4 | 1.3534 | −3.7218 | 0.4335 | 0.0009 |
| 5 | 1.3534 | −3.7219 | 0.4335 | 3×10⁻⁶ |

> Figure: The ten students. Top: the sigmoid σ(w·x + b), the boundary x* = −b/w (drag it), and each row's error pᵢ − yᵢ as a rose dash; ringed points are misclassified. Bottom left: the log-loss over (w, b) — a single convex valley, long and narrow — with the minimum marked ✕; drag the orange dot anywhere. Bottom right: each row's loss −log p(yᵢ). From w = b = 0 (loss 0.6931), one GD step (η = 0.5) reaches (0.238, 0); Newton reaches the minimum 0.4335 at (1.353, −3.722) in 4–5 clicks while GD crawls along the valley floor (it needs 7–8 clicks of GD ×50 to come within 0.0001). λ = 0.1 (sklearn C = 1) moves the minimum to (0.918, −2.523) — a flatter sigmoid with the same boundary 2.75 h.

> **Why the boundary stayed at 2.75:** This dataset is symmetric: the label pattern read from 5 h down is the mirror image (with labels flipped) of the pattern from 0.5 h up, around the mean $\bar x=2.75$. So every fit — unregularized or with any $\lambda$ — puts $p=0.5$ at exactly 2.75 h. Regularization changes only the *steepness*. In general, a penalty also moves the boundary.

## Seeing it in two dimensions

With two features the boundary is a line and the probabilities form a ramp across the plane. The player below runs plain gradient descent on 40 points: watch the background (predicted probability) sharpen and the $p=0.1/0.9$ contours pull in as $\lVert w\rVert$ grows.

> Figure: Background = predicted P(class 1); the solid line is p = 0.5, the dashed lines p = 0.1 and 0.9. Rings show each point's error pᵢ − yᵢ. Log-loss starts at ln 2 = 0.693 and the remaining mistakes sit where the classes overlap.

## The algorithm, step by step

**Fit(X, y)**

1. **Prepare.** Standardize numeric features (mean 0, variance 1) and one-hot encode categoricals; append a constant column for the intercept or keep `b` separate.
2. **Initialize** $w=0,\ b=0$ (every $p_i=0.5$; loss $=\ln 2$). The loss is convex, so the start point affects only speed.
3. **Forward.** $z=Xw+b$, $p=\sigma(z)$. Compute $\sigma$ stably: for $z<0$ use $e^{z}/(1+e^{z})$, and compute the loss from $z$ directly (e.g. $\log(1+e^{z})-yz$).
4. **Gradient.** $g_w=\tfrac1nX^\top(p-y)+\lambda w$, $g_b=\tfrac1n\sum(p_i-y_i)$.
5. **Update.** Gradient descent: $w\leftarrow w-\eta g_w$, $b\leftarrow b-\eta g_b$. Newton / IRLS: solve $(\tfrac1n\tilde X^\top S\tilde X+\lambda I')\Delta=g$ and set $\theta\leftarrow\theta-\Delta$. Quasi-Newton (L-BFGS) approximates the Hessian from past gradients.
6. **Stop** when the gradient norm (or the change in loss) falls below a tolerance `tol`, or at `max_iter`.
7. **Predict.** $p=\sigma(w^\top x+b)$; label 1 if $p\ge\tau$, with $\tau$ chosen from the costs of false positives and false negatives.

**Complexity**

- **Gradient step:** $O(nd)$ time, $O(d)$ extra memory; SGD / SAGA: $O(d)$ per row
- **Newton / IRLS step:** $O(nd^2)$ to form the Hessian + $O(d^3)$ to solve; few steps (5 on the example)
- **L-BFGS step:** $O(nd + md)$ with $m\approx10$ stored gradient pairs — scikit-learn's default
- **Prediction:** $O(d)$ per row ($O(Kd)$ for $K$ classes)

## Variants, hyperparameters and tuning

L2 strength on the worked example (scikit-learn, C = 1/(nλ) with n = 10). Stronger penalties shrink w toward 0 and raise the training log-loss; the boundary stays at 2.75 h and accuracy at 8/10.

| C | λ | w | b | Training log-loss | e^w |
| --- | --- | --- | --- | --- | --- |
| 0.1 | 1 | 0.3166 | −0.8705 | 0.5682 | 1.37 |
| 1 (default) | 0.1 | 0.9176 | −2.5233 | 0.4517 | 2.50 |
| 10 | 0.01 | 1.2765 | −3.5103 | 0.4340 | 3.58 |
| ∞ (penalty=None) | 0 | 1.3534 | −3.7219 | 0.4335 | 3.87 |

**What to tune**

- **C (or λ):** the main knob. Search on a log grid, e.g. $10^{-3}\ldots10^{3}$, by cross-validated log-loss or AUC. Note sklearn's default $C=1$ is *already* regularized — on the example it shrinks $w$ from 1.35 to 0.92.
- **penalty:** `l2` (default; smooth, keeps all features), `l1` (sparse — zeros out weak features), `elasticnet` (both; use with correlated groups of features).
- **class_weight:** `balanced` reweights rows by inverse class frequency, which shifts the intercept toward the rare class. Alternative: keep the weights and move the threshold.
- **threshold τ:** not a training parameter: pick it afterwards from a precision–recall or cost curve. If a false negative costs $c_{FN}$ and a false positive $c_{FP}$, the Bayes-optimal threshold for calibrated $p$ is $\tau=c_{FP}/(c_{FP}+c_{FN})$.
- **solver:** `lbfgs` (default, L2), `liblinear` (small data, L1), `newton-cg` / `newton-cholesky` (exact, few features), `saga` (large data, L1 / elastic net).
- **multi-class:** multinomial (softmax, one joint model — the default) or one-vs-rest (K independent binary models).
- **features:** the model is linear in what you give it: add interactions, splines or polynomial terms to bend the boundary, then regularize harder.

> **Perfect separation sends weights to infinity:** If some hyperplane separates the classes perfectly, the loss keeps falling as $\lVert w\rVert$ grows — the sigmoid tries to become a step. On the example with labels set to $[x>2.75]$, unregularized gradient descent ($\eta=1$) reaches $w=0.65$ after 10 steps, 2.09 after 100, 4.96 after 1,000 and 11.08 after 10,000, with loss heading to 0 and no end. Standard errors explode and probabilities become 0 or 1. Always keep some penalty ($C<\infty$).

> **Scale features before regularizing:** The L2 penalty treats every weight alike, so a feature measured in thousands (needing a tiny weight) is barely penalized while one measured in fractions is crushed. Unscaled features also stretch the loss valley — on the example the initial Hessian's eigenvalues differ by a factor of 53, which is why gradient descent crawls. Standardize first.

## In code

```python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def log_loss(X, y, w, b, lam=0.0):
    p = sigmoid(X @ w + b)
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)) + lam / 2 * w @ w

def fit_gd(X, y, lr=0.5, lam=0.0, tol=1e-8, max_iter=100_000):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for it in range(max_iter):
        e = sigmoid(X @ w + b) - y                 # p_i - y_i
        gw, gb = X.T @ e / n + lam * w, e.mean()
        w, b = w - lr * gw, b - lr * gb
        if max(np.abs(gw).max(), abs(gb)) < tol:
            break
    return w, b, it + 1

def fit_newton(X, y, lam=0.0, tol=1e-10, max_iter=50):
    n, d = X.shape
    A = np.c_[X, np.ones(n)]                       # last column = intercept
    theta = np.zeros(d + 1)
    R = lam * np.eye(d + 1); R[-1, -1] = 0         # never penalize b
    for it in range(max_iter):
        p = sigmoid(A @ theta)
        g = A.T @ (p - y) / n + R @ theta
        Hs = A.T @ (A * (p * (1 - p))[:, None]) / n + R
        theta -= np.linalg.solve(Hs, g)            # Newton / IRLS step
        if np.abs(g).max() < tol:
            break
    return theta[:-1], theta[-1], it + 1

hours  = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
passed = np.array([0,   0,   0,   1,   0,   1,   0,   1,   1,   1])
X = hours[:, None]

w, b, k = fit_newton(X, passed)
print(f"Newton: w={w[0]:.4f} b={b:.4f} in {k} iterations, loss={log_loss(X, passed, w, b):.4f}")
w2, b2, k2 = fit_gd(X, passed, lr=0.5, tol=1e-6)
print(f"GD:     w={w2[0]:.4f} b={b2:.4f} in {k2} iterations")
print(f"boundary x* = {-b / w[0]:.2f} h, odds ratio per hour = {np.exp(w[0]):.3f}")
print(f"P(pass | 3.8 h) = {sigmoid(3.8 * w[0] + b):.3f}")
wr, br, _ = fit_newton(X, passed, lam=0.1)
print(f"L2 lam=0.1: w={wr[0]:.4f} b={br:.4f}")     # 0.9176, -2.5233
```

From scratch: gradient descent and Newton / IRLS, with optional L2. Prints w = 1.3534, b = −3.7219 (6 Newton iterations, the last only confirming the gradient is 0; 1,255 GD iterations to gradient < 10⁻⁶), boundary 2.75 h, odds ratio 3.871, P(pass | 3.8 h) = 0.806.

```python
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss, accuracy_score

hours  = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
passed = np.array([0,   0,   0,   1,   0,   1,   0,   1,   1,   1])
X = hours[:, None]

clf = LogisticRegression(penalty=None, solver="newton-cg", tol=1e-10).fit(X, passed)
p = clf.predict_proba(X)[:, 1]
print("w =", clf.coef_[0, 0].round(4), " b =", clf.intercept_[0].round(4))
print("train log-loss =", round(log_loss(passed, p), 4), " accuracy =", accuracy_score(passed, clf.predict(X)))
print("P(pass | 1.2 h, 3.8 h) =", clf.predict_proba([[1.2], [3.8]])[:, 1].round(3))

for C in [0.1, 1, 10]:                   # C = 1 / (n * lambda)
    m = LogisticRegression(C=C, solver="newton-cg", tol=1e-10).fit(X, passed)
    print(f"C={C:<4} w={m.coef_[0, 0]:.4f} b={m.intercept_[0]:.4f} "
          f"loss={log_loss(passed, m.predict_proba(X)[:, 1]):.4f}")
```

scikit-learn on the same data. Prints w = 1.3534, b = −3.7219, log-loss 0.4335, accuracy 0.8, P(pass) = 0.109 and 0.806, then the C table above.

> **Real pipelines:** In practice wrap it as `make_pipeline(StandardScaler(), LogisticRegressionCV(Cs=20, scoring="neg_log_loss"))` so scaling is learned inside each cross-validation fold, and read probabilities with `predict_proba`, not labels with `predict`.

**Strengths**

- You need probabilities — risk scores, ranking, expected-cost decisions — and a well-calibrated baseline.
- The model must be explained or audited: $e^{w_j}$ is an odds ratio per unit of feature $j$.
- Data is large and sparse (text, one-hot categories, click logs): training is convex, fast and parallel.
- You want a strong, hard-to-overfit baseline before reaching for boosting or neural networks.

**Limitations**

- The boundary is strongly curved or driven by interactions you can't engineer — trees, boosting or kernel SVMs fit it directly.
- Features are many, highly correlated and unregularized: coefficients flip sign and stop being interpretable.
- The classes are perfectly separable and you need coefficients or p-values without a penalty — the MLE does not exist.
- Raw images, audio or text where the useful features must be learned — use a network (whose last layer is still logistic regression).

- **Clinical risk scores:** Probability of disease or readmission from age, vitals and lab values; odds ratios are the language of epidemiology.
- **Credit scoring:** Probability of default, often on binned (weight-of-evidence) features; regulators favour its transparency.
- **Ad click-through rate:** Web-scale CTR prediction on billions of sparse hashed features, trained by SGD with L1/L2.
- **Neural network heads:** The sigmoid / softmax output layer trained with cross-entropy is logistic regression on learned features.

> **Calibrated by construction:** The score equations $\sum p_i=\sum y_i$ and $\sum p_ix_i=\sum y_ix_i$ say that, on the training data, the model's average predicted probability equals the observed rate — overall and weighted by every feature. That is why logistic regression probabilities are usually well calibrated out of the box, while SVM scores or boosted-tree margins often need Platt scaling — which is itself a one-feature logistic regression.

> **Accuracy hides imbalance:** With 1% positives, predicting all-negative scores 99% accuracy. Judge the model with log-loss, ROC-AUC or precision–recall, and choose the threshold from the costs of errors, not 0.5 by default.

**Question:** In the worked example $w=1.3534$ per hour. Studying one more hour multiplies the **odds** of passing by about…

- A. 1.35
- B. 3.87
- C. 0.81
- D. it depends on how many hours you already studied

**Answer:** B. 3.87 — Odds multiply by $e^{w}=e^{1.3534}=3.87$ per unit, whatever the starting point. It is the *probability* change that depends on the starting point.

**Question:** A model has $w=2$ and $b=-6$ on one feature. Where is the decision boundary at threshold 0.5?

- A. x = 0.5
- B. x = 3
- C. x = −3
- D. x = 6

**Answer:** B. x = 3 — $p=0.5\iff z=0\iff 2x-6=0\iff x=-b/w=3$.

**Question:** You fit an unregularized logistic regression to data that a straight line separates perfectly. What happens?

- A. It finds the maximum-margin line, like an SVM
- B. The weights grow without bound as training continues
- C. The loss gets stuck at ln 2
- D. It converges to the same weights as least squares

**Answer:** B. The weights grow without bound as training continues — Scaling $w$ up always lowers the loss on separable data, so no finite minimizer exists: on the example, $w$ went 0.65 → 2.09 → 4.96 → 11.08 over 10 → 10,000 steps. A penalty ($C<\infty$) restores a finite optimum.

**Question:** After fitting the worked example (with an intercept), what is $\sum_i p_i$ over the 10 training students?

- A. 0.5
- B. 4.335
- C. 5
- D. it can be any value

**Answer:** C. 5 — At the optimum $\partial L/\partial b=\frac1n\sum(p_i-y_i)=0$, so $\sum p_i=\sum y_i=5$ — the number of students who passed.

**Key points**

- $p=\sigma(w^\top x+b)$: the log-odds are linear; the boundary is the hyperplane $w^\top x+b=0$; $e^{w_j}$ is the odds ratio.
- Train by minimizing cross-entropy (the negative log-likelihood). Its gradient is $\tfrac1nX^\top(p-y)$ — sigmoid and log cancel.
- The Hessian $\tfrac1n\tilde X^\top S\tilde X$ is positive semi-definite, so the loss is convex; Newton / IRLS converges in a handful of steps, gradient descent in hundreds.
- Regularize ($C=1/(n\lambda)$; L2 shrinks, L1 sparsifies) — essential when classes are separable or features many and correlated. Scale features first.
- Use probabilities, not labels: evaluate with log-loss / AUC and set the threshold from error costs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
