Linear Regression

Linear regression predicts a number as a weighted sum of features. It’s simple, fast and interpretable, and it introduces every idea that bigger models reuse.

The model

For all $m$ examples at once: $\hat{\mathbf{y}} = X\mathbf{w} + b$. Each weight $w_j$ is the change in the prediction when feature $j$ increases by one unit, holding the others fixed.

Example: price (₹ lakh) = 0.05 × area (sq.ft) + 4 × bedrooms − 0.3 × age + 10. A 1,000 sq.ft, 2-bedroom, 10-year-old flat → 50 + 8 − 3 + 10 = ₹65 lakh.

The loss: mean squared error

Squaring punishes big errors heavily and makes the loss smooth and convex: one global minimum. (It’s also the maximum-likelihood loss under Gaussian noise.)

Solving it

Closed form (normal equation): set the gradient to zero:

It’s exact and needs no learning rate, but it costs about $O(n^3)$ to invert and fails if $X^\top X$ is singular (duplicate or collinear features).

Gradient descent: repeat

It scales to millions of examples and features. Standardise features first, $x' = (x - \mu)/\sigma$, or the loss bowl is stretched and descent is slow.

Beyond straight lines

“Linear” means linear in the weights, not the inputs. Add transformed features such as $x^2$, $\log x$ or $x_1 x_2$ and you can fit curves (polynomial regression). More features means more flexibility, and more risk of overfitting: a degree-15 polynomial can thread through every training point and be useless in between.

flowchart LR
  A["Degree 1:<br/>underfits"] --- B["Degree 3:<br/>good fit"] --- C["Degree 15:<br/>overfits"]

Evaluating

• MSE / RMSE (root of MSE, in the target’s units): “typical error ±₹4 lakh”.
• MAE (mean absolute error): less sensitive to outliers.
• R² $= 1 - \frac{\sum (y_i - \hat y_i)^2}{\sum (y_i - \bar y)^2}$: the fraction of variance explained. 0 means no better than predicting the mean; it can be negative on test data.

Assumptions (and what breaks)

1. Linearity: the relationship is linear in the features. If not, add features or use another model.
2. Independent errors: violated by time series, where neighbouring errors correlate.
3. Constant error variance (homoscedasticity): errors that grow with the prediction suggest transforming the target, e.g. $\log(\text{price})$.
4. No strong multicollinearity: correlated features make coefficients unstable.
5. Outliers pull hard on squared error; consider MAE, Huber loss, or cleaning the data.

Regularised versions

• Ridge (L2): add $\lambda \lVert \mathbf{w}\rVert_2^2$. It shrinks weights, stabilises against collinearity, and has the closed form $(X^\top X + \lambda I)^{-1}X^\top\mathbf{y}$.
• Lasso (L1): add $\lambda \lVert\mathbf{w}\rVert_1$. It drives some weights exactly to zero, which performs feature selection.
• Elastic Net: both.

More on these in the bias–variance lesson.