Transformers and how a language model writes

On 12 June 2017 eight researchers posted a paper to arXiv with a title that sounded like a slogan: “Attention Is All You Need”. It proposed a network for translation, the Transformer, built entirely on a mechanism called attention, with no recurrence and no convolutions.

It translated English into German better than any previous model, ensembles included, scoring 28.4 BLEU, and reached 41.8 BLEU on English to French after three and a half days of training on eight GPUs. The architecture turned out to be good at far more than translation.