Cloudflare CEO says global outage triggered by internal error, not cyberattack

Cloudflare CEO Matthew Prince has confirmed that the widespread internet disruption that affected platforms across the world was caused by an internal configuration error rather than a cyberattack.

The outage, which briefly brought down services such as X, ChatGPT, Canva, Discord and numerous other websites and applications, was traced to a permissions update on a ClickHouse database cluster. The change, intended to streamline access to data, inadvertently triggered a faulty query that caused the system to extract significantly more data than intended.

As a result, a crucial “feature file” used by Cloudflare’s Bot Management system began expanding abnormally. This file, which is regenerated and distributed throughout Cloudflare’s global network every five minutes, suddenly exceeded the software’s size limits. The oversized file caused routing software at the network’s edge to crash, leading to recurrent failures.

Complicating matters further, the faulty file was generated only on the parts of the database cluster that had been updated. This created a cycle in which Cloudflare’s network would briefly recover when a correct file was distributed, only to fail again minutes later when a corrupted version propagated. The disruption lasted for nearly three hours from around 11:20 UTC.

Prince acknowledged that Cloudflare’s initial assessment mistakenly pointed to a massive DDoS attack, owing to the pattern of failures. Engineers later isolated the issue, halted the spread of the corrupted file, restored an older valid version, and rebooted affected systems. Services were fully stabilised by 17:06 UTC, with the company describing the incident as its most serious outage since 2019.

Apologising for the disruption, Prince said Cloudflare would implement stronger safeguards to prevent similar incidents, including stricter file-size controls, global kill switches for critical updates, and a broader review of the resilience of its core infrastructure.

With IANS Inputs