All news with #clickhouse tag

Cloudflare outage (18 Nov 2025): feature file duplication

⚠️ On 18 November 2025, Cloudflare experienced a major outage after a permissions change in a ClickHouse database caused duplicated metadata to be emitted into a Bot Management feature file, doubling its size. The oversized file exceeded a preallocated feature limit in the core proxy, triggering a Rust panic and widespread HTTP 5xx errors. Cloudflare halted propagation, restored a known-good file, and restarted the proxy; services were largely restored by 14:30 UTC and fully recovered by 17:06 UTC. The company apologized and pledged architectural and process hardening to prevent recurrence.