Evaluation and cross-validation

In 1974 the statistician Mervyn Stone read a paper to the Royal Statistical Society titled “Cross-Validatory Choice and Assessment of Statistical Predictions”. His point was that the same data cannot honestly be used both to fit a model and to judge it; but if you leave part out, fit on the rest and rotate, every observation can serve as a test case once.

Stone also separated two jobs that people still mix up: choosing a model (or its hyperparameters) and assessing the one you chose. Doing both with the same held-out data makes the reported score optimistic.