All news with #postmortem tag

🛠️ Cloudflare’s Images service, built in Rust on Workers, encountered intermittent truncation of larger transformed images after a 2025 rearchitecture that connected Workers to Images via a local binding. The failure returned HTTP 200 with a shortened body and no logged errors, appearing only under production concurrency and reproducible as a timing-dependent race in the hyper HTTP library. Instrumentation, strace, and controlled reproductions identified premature shutdown calls from the Images service that left most response bytes in hyper’s internal buffer; a four-line change ultimately fixed the issue.

🔒 The State of Nevada published a detailed after-action report describing how attackers used a trojanized system administration utility to establish persistent access and deploy ransomware across state infrastructure. The initial compromise occurred on May 14 and was detected on August 24, impacting more than 60 agencies and prompting a 28-day recovery that restored 90% of required data without paying a ransom. Nevada engaged external responders including Microsoft DART and Mandiant, and has since implemented account cleanups, password resets, certificate removals, and tightened access controls.

🏛 Thieves brazenly used a furniture elevator to access a second‑floor window and stole historic jewels worth about €88 million from display cases at the Louvre in October 2025. French authorities say the alarms on the affected window and cases functioned as intended, but the theft prompted a comprehensive security review and urgent recommendations for new governance, extra perimeter cameras, and updated protocols. Confidential audits cited by Libération document chronic IT weaknesses since 2014 — systems running Windows 2000 and weak password hygiene, including a video server reportedly protected by the password "LOUVRE".

🔒 Markus Weber, founder and managing director of dokuworks, describes the immediate steps his team takes when called in after a cyberattack: isolate and secure affected systems so IT forensics can operate, preserve extortion correspondence to help identify perpetrators, assess operational impact, and initiate emergency operations. He warns that ransomware is the predominant threat and generally advises against paying ransoms, though there are rare exceptions. Many organizations are improving technically but still neglect documented emergency organization and trusted external partnerships, leaving them vulnerable.

⚠️ A bug in a dashboard React useEffect dependency caused an object to be recreated on every render, triggering repeated calls to the Tenant Service /organizations endpoint. Those excessive requests coincided with a Tenant Service deployment, overwhelming the service and breaking API authorization checks so many API requests returned 5xx errors and the Cloudflare dashboard became unavailable. Cloudflare mitigated the incident by scaling pods, applying a global rate limit, reverting a problematic patch, and applying a dashboard hotfix. They plan to prioritize Argo Rollouts for safer deployments, add randomized retry delays, increase Tenant Service capacity, and improve observability.