Data quality, bias and leakage

In 2014 a team at Amazon began building software to rate job applicants from one to five stars, much as shoppers rate products. Its models learned from résumés submitted to the company over about ten years, and most of those had come from men.

The system learned that pattern. According to people familiar with the project, reported by Reuters in October 2018, it penalised résumés containing the word “women's” and downgraded graduates of two all-women's colleges. Amazon edited the programs to be neutral to those terms, but that was no guarantee the models would not find other ways to discriminate, and the team was disbanded by the start of 2017. Recruiters had looked at its recommendations but, the sources said, never relied on them alone.

Nobody told the model to prefer men. It was told to imitate past decisions, and it did.