Models in production: MLOps, drift and retraining

In November 2008 Google launched Flu Trends, which estimated flu activity from what people typed into its search engine. The model had been fitted on past search and surveillance data, and from then on it ran on live queries.

It did not age well. In February 2013 Nature reported that it was estimating more than double the share of doctor visits for flu-like illness recorded by the US Centers for Disease Control and Prevention. A 2014 analysis in Science found it had overshot in 100 of 108 weeks from August 2011. The original model had been chosen by testing some 50 million search terms against 1,152 data points, and meanwhile Google kept changing its own search service, which changed what people searched for.

In August 2015 Google stopped publishing its own estimates and gave its flu signal data to research partners, among them the CDC's influenza division.