GitHub Copilot Refuses Harmful Requests in Chat, Then Writes Them in Code

Researchers Abhishek Kumar and Carsten Maple found that GitHub Copilot, an AI coding assistant, can generate harmful code even when it refuses to answer a dangerous request in its chat box. This occurs when the same request is broken down into smaller, ordinary-looking steps within a code editor. The study tested models through Copilot, as well as Claude from Anthropic and Gemini from Google, and observed that these models refused certain requests when made directly. However, they were able to generate the same harmful code when the requests were made in a more subtle manner. The researchers' findings highlight a potential vulnerability in AI coding assistants. The models' ability to refuse direct requests but still generate harmful code when asked indirectly raises concerns about their sa