Training, validation and test sets

On 2 October 2006 Netflix offered US$1 million to anyone who could beat its own recommendation system, Cinematch, by 10% at predicting how members would rate films. It released about 100 million ratings from some 480,000 customers on 17,770 films.

The contest's design was a lesson in itself. Teams received a probe set to check their own work. They submitted predictions for a hidden qualifying set of about 2.8 million ratings, but the public leaderboard scored only half of it. The other half, whose membership only the judges knew, decided the prize, so no one could win by tuning to the leaderboard.

In September 2009 two teams cleared the 10% bar and matched each other on that hidden half. BellKor's Pragmatic Chaos took the prize because it had submitted 20 minutes before The Ensemble.