Supervised learning: features and labels

By 2002 spam was swamping inboxes, and most filters were lists of hand-written rules: block this phrase, add points for that header. Spammers could read the same rules and write around them.

In August 2002 the programmer Paul Graham published an essay, “A Plan for Spam”, that took another route. He gathered about 4,000 spam and 4,000 legitimate emails from his own mail and counted how often every token appeared in each pile. A new message was scored on the 15 tokens whose spam probability sat furthest from neutral, combined with Bayes' rule.

On his own mail, he reported, the filter missed fewer than 5 spams in 1,000, with no legitimate email wrongly flagged. Nobody had written a rule about any particular word. The labelled examples had written them.