Deep Dive: Cloudflare's Sept 12 Dashboard and API Outage
⚠️ A bug in a dashboard React useEffect dependency caused an object to be recreated on every render, triggering repeated calls to the Tenant Service /organizations endpoint. Those excessive requests coincided with a Tenant Service deployment, overwhelming the service and breaking API authorization checks so many API requests returned 5xx errors and the Cloudflare dashboard became unavailable. Cloudflare mitigated the incident by scaling pods, applying a global rate limit, reverting a problematic patch, and applying a dashboard hotfix. They plan to prioritize Argo Rollouts for safer deployments, add randomized retry delays, increase Tenant Service capacity, and improve observability.
