Neural networks and backpropagation

In 1970 Seppo Linnainmaa, a student at the University of Helsinki, wrote a master's thesis on estimating the rounding errors that pile up inside long computations. To do it he needed the derivative of a final result with respect to every intermediate quantity, and he found a way to get all of them in one sweep backwards through the computation.

That method, now called reverse-mode automatic differentiation, is exactly what training a neural network requires: the derivative of one number, the loss, with respect to millions of weights. Applied to networks and popularised in 1986, it took the name backpropagation.