Linear regression and gradient descent

In 1805 Adrien-Marie Legendre, working out the orbits of comets, added an appendix to his book describing the method of least squares: choose the parameters that make the sum of squared residuals as small as possible. Gauss later said he had used it since the 1790s, and the two men argued over priority for years.

Least squares has an exact solution, but solving it for many unknowns is expensive. In 1847 Augustin-Louis Cauchy, also motivated by astronomical calculations, proposed a different route: move the unknowns a little in the direction that decreases the error fastest, and repeat. That is gradient descent, the procedure that trains almost every model today.