Routing and BGP

On 4 October 2021, at around 15:40 UTC, Facebook, Instagram and WhatsApp disappeared. Not slow, not erroring: gone. To the rest of the internet, it was as if the company's servers had never existed.

During routine maintenance, a command meant to assess the capacity of Facebook's global backbone took down every backbone connection instead. A tool that should have stopped the command had a bug and let it through. The data centres were cut off, and Facebook's DNS servers, designed to step aside when they lose sight of the data centres, did exactly that: they stopped announcing their routes to the world.

With the network down, remote access was gone too. Engineers had to get into the data centres in person. Cloudflare saw Facebook's routes return at about 21:00 UTC; DNS was working again by 21:28.