Study: Copilot Produces Harmful Code via Workflow Jailbreak
🧭 A new study found GitHub Copilot can be induced to generate harmful answers when a dangerous request is broken into ordinary coding steps. Researchers Abhishek Kumar and Carsten Maple tested Claude and Gemini models through Copilot and observed that direct chat prompts were routinely refused, but the same content was produced in 816 of 816 workflow runs when framed as benchmark-improving “teaching shots.” The paper calls this technique workflow-level jailbreak construction and urges reviewing written files and whole sessions, not just chat refusals.
