What Happened During The March 2018 Facebook Outage
I was on call for a smaller cloud provider around that time, and I watched the incident unfold on Twitter because our monitoring hadn't yet gone dark. Within forty-five minutes, Facebook's infrastructure had collapsed completely. Not degraded. Not slow. Gone. The entire network simply disappeared from the internet because of a single configuration command that shouldn't have been able to do what it did. The official root cause was straightforward enough after the fact. Facebook was performing routine changes on their backbone core routers, modifying the BGP configuration. Someone ran a command that filtered outbound route advertisements, and Facebook announced zero routes for itself. Every major ISP on the planet then removed Facebook's autonomous system from their routing tables. People couldn't reach facebook.com, scontent.facebook.com, or any of the CDN endpoints. The outage lasted roughly eleven hours before engineers at Menlo Park manually re-uploaded the correct route announcements and waited for global convergence. But the real story is in the details that matter if you ever have to build something this big, and the headline here is that most organizations don't have the safeguards that should have prevented this.
The Tragic Tale Of The Parallelogram That Broke Facebook A Case Study In Algorithmic Chaos
The parallelogram reference comes from how the routing topology behaved during the incident. When Facebook withdrew all its BGP routes, the network didn't just go dark uniformly. There was a cascading effect where certain peering points lost connectivity first, then others dropped as upstream providers processed the withdrawal announcements. Engineers described the collapse pattern as having a shape — routes vanished along one edge of the topology and propagated across in a predictable cascade. It's not a formal geometry term, just a way to visualize the asymmetric way the failure spread through their backbone. Here's what I found more disturbing than the outage itself: the command that triggered it was not some exotic, high-risk change. It was a standard route filter update. Something your junior network engineer might push on a Tuesday afternoon. And it took down one of the largest platforms on the internet.
Why BGP This Breaks Things That Shouldn't Be Able To Break
BGP — the Border Gateway Protocol — is the routing system that tells the internet how to reach different networks. It's trust-based by design. When a network announces routes, other networks generally accept them. There's no universal validation layer. If a router says "I can reach these prefixes," the world takes its word for it until updated information arrives. Facebook's issue was specifically a route withdrawal, not a route injection. Their autonomous system number (AS32934) simply stopped advertising reachability to any of its IP prefixes. ISPs worldwide processed the withdrawal and removed Facebook from their routing tables. The effect was instantaneous and global because BGP updates propagate quickly, and every upstream provider was syncing the same bad news. The counter-intuitive thing about BGP is that the protocol is deliberately simple. That's also its danger. A single misconfigured access control list on a single router can nullify an entire autonomous system's presence on the internet. There's no safety net baked into the protocol itself.
Get the Full Details
The Technical Mechanics Of The Failure
Facebook uses an internal routing system called BOLT (Bidirectional Online Topology). It computes shortest-path trees across their network and programs the forwarding tables on their core routers. BOLT generates the BGP advertisements that tell the outside world how to reach Facebook's infrastructure. During the routine change, the team was modifying route policy on the backbone routers. The command modified outbound route filtering, effectively telling the routers to stop advertising Facebook's prefixes to their BGP peers. BOLT continued running normally internally, but the external advertisements carried zero valid routes. Here's the part that catches people: BOLT itself didn't fail. The internal routing was fine. The problem was purely in the translation layer between BOLT's internal decisions and the BGP advertisements sent to the outside world. This distinction matters because it means the monitoring systems that track internal network health showed nothing wrong. The outage was invisible to most of Facebook's own tooling until external services started reporting failures.
What This Means For Your Infrastructure
If you're running anything with external-facing routing, here's what you need to understand. Most teams configure BGP in a way that assumes human oversight catches mistakes. That's a bad assumption. The first thing to implement is route leak prevention using RPKI (Resource Public Key Infrastructure). RPKI allows you to cryptographically validate that an AS is authorized to announce specific IP prefixes. Without it, any misconfigured router can accidentally or deliberately announce routes it shouldn't. I've seen this exact problem bite a mid-size hosting provider when a single misconfigured upstream peer started advertising /24 prefixes that belonged to someone else, causing partial traffic blackholes that took six hours to diagnose. The second thing is conservative change management. Before pushing BGP route modifications, run simulated changes against a test topology or use BGP route dampening and hold timers to give yourself a rollback window. Facebook's engineers couldn't roll back fast because the announcement had already propagated globally, and convergence takes time even when you push the fix.
A Workaround I Learned The Hard Way
When I was building out routing for a multi-tenant platform, we had a near-miss that reminded me of the Facebook incident. I was deploying a new upstream provider and made a mistake in the prefix filtering that caused us to withdraw half our advertised routes. The change happened at 2:14 AM on a weekend. The workaround I implemented immediately after was a pre/post-change validation script that checks three things before and after any BGP modification: the outgoing route count on each BGP session, the AS-path length for all advertised prefixes, and a comparison against a known-good baseline stored in version control. The script runs automatically before the change is applied and again thirty seconds after. If the post-change numbers deviate by more than five percent from the baseline, the script alerts immediately and recommends rollback. This cut our mean time to detection from something like two hours to under four minutes in subsequent incidents. The script itself took about three days to write and refine. Worth every hour.
The Limitations You Should Know About
RPKI adoption is still incomplete globally. As of my last check, only a fraction of the world's routers validate RPKI-originated routes. Facebook's incident happened before RPKI was widely deployed in the paths that matter, and even today many ISPs don't enforce strict RPKI filtering on incoming routes. Don't assume RPKI alone will save you. BGP monitoring tools like Looking Glass instances and route collectors (Route Views, RIPE RIS) can give you visibility into how your routes are perceived externally, but there's typically a thirty-to-sixty-minute delay between a route change and when those systems reflect it. If your outage happens and you're relying on public route collectors to tell you what's wrong, you're already behind. The most honest assessment is that BGP was designed in an era when the internet was a small trusted network. It doesn't scale to the current environment where a single misconfiguration can take down major services. The safeguards exist, but they're largely optional and underutilized.
What To Do If You're Responsible For This Stuff
Start with change controls. Every BGP modification should go through a review process, even if it's just a Slack comment from a peer. Implement automated validation scripts. Use RPKI where your upstream providers support it. Build monitoring that specifically tracks BGP session state and route count deltas, not just uptime ping checks. The Facebook outage wasn't tragic because of complexity. It was tragic because a routine change on a well-tested system took down a critical internet service, and the safeguards that should have prevented it weren't in place. The same thing can happen to anyone running BGP, regardless of scale.
