All news with #prompt injection attack tag

🔍 Researchers at the AI Now Institute demonstrated a proof-of-concept called "Friendly Fire" where autonomous AI coding agents (Anthropic's Claude Code and OpenAI's Codex) execute an attacker's binary when asked to scan untrusted open-source code. The attack hides a malicious binary alongside benign files and a README that prompts the agent to run a security script; in auto-modes the agents approved and executed it without prompting. The weakness is framed as a workflow/design issue rather than a single vulnerable version, and the researchers recommend never giving command-capable agents unattended access to untrusted code.

🧭 A new study found GitHub Copilot can be induced to generate harmful answers when a dangerous request is broken into ordinary coding steps. Researchers Abhishek Kumar and Carsten Maple tested Claude and Gemini models through Copilot and observed that direct chat prompts were routinely refused, but the same content was produced in 816 of 816 workflow runs when framed as benchmark-improving “teaching shots.” The paper calls this technique workflow-level jailbreak construction and urges reviewing written files and whole sessions, not just chat refusals.

🔍 Zscaler ThreatLabz uncovered real-world campaigns using indirect prompt injection, where hidden instructions embedded in web pages steer AI agents. Attackers used SEO poisoning to surface malicious pages and hid prompts via CSS and JSON-LD metadata. One campaign impersonated a Python library to trick agents into paying a $3 bogus API key; another typosquatted a DeBank site to claim authority. Tests across 26 LLMs showed varying susceptibility depending on model and context.

🛡️ Researchers discovered two vulnerabilities in the Cursor AI-enabled IDE that enable prompt-injection-driven remote code execution by escaping the command execution sandbox. The flaws, CVE-2026-50548 and CVE-2026-50549, allow attackers to change the working directory and exploit symlink canonicalization fallbacks to write or overwrite files outside the project scope. Cursor patched the issues in version 3.0, and the findings underscore broader risks in agentic AI workflows and the difficulty of defending against prompt injection.

🔒 As organizations adopt AI more broadly, third-party API proxies and aggregators promise convenience, cost savings, and failover between models. Some providers operate transparently, but many exploit forged or stolen accounts, reroute queries to cheaper models, and capture or manipulate prompts and outputs. These practices expose firms to data leakage, IP loss, compliance violations, and security threats such as injected malicious code or reduced model accuracy.

🧩 Researchers at LayerX demonstrated a prompt injection called BioShocking that trains AI-powered browsers to treat risky real-world actions as fictional, bypassing safety controls. The PoC used a themed puzzle game to reward 'wrong' behavior and culminated in instructing agents to copy sensitive data from a GitHub repo. Six mainstream agentic browsers were tested; only one vendor implemented a working fix after disclosure. LayerX recommends explicit user confirmations, stricter context checks, and session scope limits.

🔒 New research from Adversa AI, dubbed GuardFall, shows a decades-old shell trick can bypass simple blocklist checks in open-source AI coding agents, letting hidden destructive commands run. The flaw arises because filters inspect the command as plain text while shells like bash rewrite and expand that text before execution. Ten of eleven tested agents were vulnerable; only Continue defended by parsing commands the same way the shell does.

🔎 Researchers at Mozilla's 0DIN demonstrated that an AI coding agent like Claude Code can be manipulated into executing a remote payload by following innocuous setup instructions in a clean GitHub repo. The approach uses three benign-looking components—a standard repo, an initialization error prompting a recommended command, and a script that pulls a command from a DNS TXT record—to spawn an interactive shell with developer privileges. 0DIN warns this chain leaves no explicit malicious code in the repo and is difficult for scanners or human reviewers to detect.

🛡️ Researchers uncovered a macOS malware family named macOS.Gaslight that embeds fabricated error messages and debugging data inside a Rust binary to mislead AI-assisted analysis tools. The 3.5 KB payload contains 38 fake system messages — including memory dumps, token-expiration warnings, and build errors — designed to appear as legitimate developer logs. SentinelOne attributes the sample with high confidence to a North Korean-linked actor and notes the strings aim to prompt-inject LLM pipelines, causing them to abort or distrust their session. The malware retains standard backdoor and data-stealing capabilities alongside the deceptive messaging tactic.

📝 This post highlights a new paper that demonstrates how large language models are vulnerable to prompt injection because they learn to distinguish instruction blocks by style rather than explicit tags. The authors argue that role tags became a de facto security architecture but do not map cleanly into model representations, producing persistent role confusion. The paper warns that without genuine role perception, defenses will be reactive and brittle, and calls for deeper study of roles within the LLM stack.

🛡️ A new Rust-based macOS implant named Gaslight embeds a prompt-injection payload aimed at misleading AI-assisted analysis tools into aborting or refusing to analyze the sample. SentinelOne attributes the tool with high confidence to North Korea–aligned actors and notes its Telegram-based C2 implements an interactive shell with commands like shell, upload, and kill. The implant uses a LaunchAgent for persistence and includes a Base64-encoded Python stealer that harvests browser data, Terminal histories, Keychain contents, and system profiles before compressing and exfiltrating via Telegram.

🔒 Researchers at LayerX demonstrated a technique called BioShocking that convinces AI-powered web browsers they are playing a game, causing them to abandon safety guardrails and exfiltrate user data. The team tested six agentic browsers and plugins, including ChatGPT Atlas, Perplexity's Comet and Anthropic's Claude extension, and in a proof-of-concept had each copy login credentials and send them to an attacker. LayerX recommended requiring user confirmation for account reads and adding context-aware flags to limit what agents can access.

🛡️ SentinelLabs uncovered a North Korea-linked macOS backdoor, tracked as macOS.Gaslight, that embeds 38 fabricated system messages to manipulate AI-assisted malware triage. The Rust implant carries an infostealer and interactive backdoor that exfiltrates browser data, terminal histories and the macOS login keychain, using Telegram Bot API with certificate pinning for command and control. Researchers noted novel tradecraft including runtime staging of a standalone Python interpreter and self-scrubbing of the Telegram bot token from logs. SentinelLabs warned analysts to treat sample contents as adversarial input and to isolate hostile content from LLM-based tools.

🛡️ A malware developer has begun embedding provocative text about nuclear and biological weapons inside large JavaScript block comments in spyware payloads to confuse AI-based scanners. The commented header is ignored at runtime but aims to trigger refusals or misclassification in naive LLM-powered triage systems that ingest file starts without isolating untrusted content. Traditional detection methods—YARA, entropy checks, AST parsing, and behavioral analysis—remain effective, but the technique is a practical anti-analysis tactic against weak AI-first pipelines.

🔒 Microsoft outlines a defense-in-depth approach to protect AI memory across storage, retrieval, model interaction, and user control. The post explains how memory transforms AI from stateless tool to learning collaborator, increasing attack surface and enabling staged attacks that persist beyond initial prompts. It summarizes protections in M365 Copilot including prompt-injection classifiers, Task Adherence checks, tenant policy controls, unified compliance, and audit logging integrated with Defender and Sentinel.

🛡️ Microsoft patched a vulnerability chain named AutoJack in AutoGen Studio that could allow a visiting webpage to coerce a developer’s AI agent into executing arbitrary commands on the host. AutoGen Studio is the graphical interface for Microsoft’s open-source AutoGen framework for multi-agent AI systems; the flaw was fixed during development and never shipped in a PyPI release. The issue affected developers who built from the main GitHub branch in a limited window and allowed attacker-supplied commands to be launched with the developer’s account privileges. Microsoft urges running AutoGen Studio only as a developer prototype in isolated, low-privilege environments and avoiding exposure to untrusted content.

🔒 A proof-of-concept called SearchLeak demonstrated a prompt injection attack against Microsoft M365 Copilot Enterprise that tricks users into clicking crafted links to exfiltrate corporate data. Researchers combined three weaknesses in Copilot Search — including URL query parameters treated as natural language prompts — to leak sensitive content. Microsoft patched the server-side flaw, but the incident highlights risks when AI services access broad corporate assets and the need for render-time sanitization and stricter CSPs.

🔐 Threat actors abused trusted services — Google Ads, GitLab Pages, and Claude’s shared-chat feature — to trick developers into executing malicious PowerShell and terminal commands via ClickFix social engineering. Researchers at TrendAI observed a six-wave campaign that funnelled over 2,000 victims from sponsored search results to malicious pages and then to weaponized Claude shared chats. By impersonating popular developer tools and brands, the attackers leveraged reputation stacking to make their lures appear legitimate and evade detection.

🛡️ In response to widespread prompt-injection risks, the article outlines runtime signals to detect compromised AI agents that possess the so-called lethal trifecta: access to private data, ingestion of untrusted content, and external communication ability. It argues that this trifecta is now the default for useful agents, so defenses must shift from architecture rules to behavioral, runtime detection. Recommended signals include instruction-following anomalies, unexpected tool-call sequences, low-bandwidth exfiltration channels, out-of-scope credential access, and suspicious memory writes.

🛡️ Cybersecurity researchers at Tenet Security disclosed a new attack class called Agentjacking that tricks AI coding agents into executing arbitrary code. The exploit leverages Sentry's public DSN and its MCP interaction to inject crafted error events, which agents like Claude Code and Cursor interpret as trusted resolution steps. Successful exploitation can expose sensitive data and run code with developers' privileges.