Global outage was caused by internal error, not cyberattack: Cloudflare CEO
New Delhi, Nov 19
Cloudflare CEO Matthew Prince has confirmed that the major global internet outage was not the result of a cyberattack, but was caused by an internal configuration mistake.
The outage disrupted several major platforms, including X, ChatGPT, Canva, Discord, and many other websites and apps around the world.
In a postmortem, Prince explained that the problem began when the company made a change to permissions on a ClickHouse database cluster.
The update was meant to improve access to data, but a faulty query caused the system to pull in far more information than it should have.
This error made a key “feature file†used by Cloudflare’s Bot Management system grow too large.
This feature file is refreshed and shared across Cloudflare’s network every five minutes. When the file suddenly doubled in size, it crossed the software’s limit, causing routing software at the network edge to crash.
The issue became unpredictable because the faulty file was only generated on parts of the cluster that had been updated. This meant that every five minutes, Cloudflare’s network either received a correct file and briefly recovered or a corrupted file and failed again.
This cycle of recovery and failure continued for about three hours from around 11:20 UTC, causing widespread service interruptions across the internet. Prince stressed that there was no cyberattack involved and admitted that the company initially misread the symptoms as a massive DDoS attack before identifying the real cause.
Engineers eventually stopped the faulty file from spreading, replaced it with an older correct version, and restarted the affected systems. Cloudflare said the issue was fully resolved by 17:06 UTC and described the incident as its most serious outage since 2019.
Prince apologised for the disruption and said Cloudflare will introduce stronger safeguards, including stricter limits on file sizes, global kill switches for critical updates, and a broader review of how its core systems can fail.
— IANS
Reader Comments
At least they're being transparent about it. Many companies would try to hide such internal errors. The 3-hour cycle of recovery and failure must have been frustrating for users worldwide 😅
As a software engineer myself, I can understand how these configuration mistakes happen. But for a company of Cloudflare's scale, there should be better safeguards. The global kill switch idea is good - should have been there from the start.
I was trying to submit my project on Canva when this happened! Thought it was my internet connection and kept restarting the router 😂 Now I know it was a global issue. Hope they implement those safeguards quickly.
This shows how interconnected our digital world has become. One company's internal error can disrupt services globally. Makes you think about having backup systems in place for critical business operations.
While I appreciate the transparency, this level of outage affecting so many major platforms is concerning. Companies like Cloudflare need to have more robust testing environments before rolling out changes to production systems. The apology is good, but prevention is better.
We welcome thoughtful discussions from our readers. Please keep comments respectful and on-topic.