The bias–variance trade-off and regularisation

In 1970 Arthur Hoerl and Robert Kennard, working at DuPont, published a paper in Technometrics with a provocative title: “Ridge Regression: Biased Estimation for Nonorthogonal Problems”. When predictors are strongly correlated, the least-squares coefficients become huge, unstable and change sign from one sample to the next.

Their remedy was to accept a little bias on purpose, adding a penalty that shrinks the coefficients, in return for a large drop in variance. Statisticians had been taught that an unbiased estimator was the virtuous one. Hoerl and Kennard showed that on average a biased one could be closer to the truth.