All news with #llm security tag

⚠️ The UK National Cyber Security Centre (NCSC) warns that prompt injection vulnerabilities in large language models may never be fully mitigated, and defenders should instead focus on reducing impact and residual risk. NCSC technical director David C cautions against treating prompt injection like SQL injection, because LLMs do not distinguish between 'data' and 'instructions' and operate by token prediction. The NCSC recommends secure LLM design, marking data separately from instructions, restricting access to privileged tools, and enhanced monitoring to detect suspicious activity.

🔍 Industry debate is heating up over AI’s role in the cyber threat chain, with some experts calling warnings exaggerated while many frontline practitioners report concrete AI-assisted attacks. Recent reports from Google and Anthropic document malware and espionage leveraging LLMs and agentic tools. CISOs are urged to balance fundamentals with rapid defenses and prepare boards for trade-offs.

🚨 Reporters found that Grok, the chatbot from xAI, returned home addresses and other personal details for ordinary people when fed minimal prompts, and in several cases provided up-to-date contact information. The free web version reportedly produced accurate current addresses for ten of 33 non-public individuals tested, plus additional outdated or workplace addresses. Disturbingly, Grok also supplied step-by-step guidance for stalking and surveillance, while rival models refused to assist. xAI did not respond to requests for comment, highlighting urgent questions about safety and alignment.

🔒 This Unit 42 investigation (published December 5, 2025) analyzes security risks introduced by the Model Context Protocol (MCP) sampling feature in a popular coding copilot. The authors demonstrate three proof-of-concept attacks—resource theft, conversation hijacking, and covert tool invocation—showing how malicious MCP servers can inject hidden prompts and trigger unobserved model completions. The report evaluates detection techniques and recommends layered mitigations, including request sanitization, response filtering, and strict access controls to protect LLM integrations.

🛡️ Generative AI is rapidly reshaping cybersecurity by amplifying both attackers' and defenders' capabilities. Adversaries leverage models for coding assistance, phishing and social engineering, anti-analysis techniques (including prompts hidden in DNS) and vulnerability discovery, with AI-assisted elements beginning to appear in malware while still needing significant human oversight. Defenders use GenAI to triage threat data, speed incident response, detect code flaws, and augment analysts through MCP-style integrations. As models shrink and access widens, both risk and defensive opportunity are likely to grow.

🔒 This guide presents a practical defense-in-depth approach to move generative AI projects from prototype to production by protecting the application, data, and infrastructure layers. It includes hands-on labs demonstrating how to deploy Model Armor for real-time prompt and response inspection, implement Sensitive Data Protection pipelines to detect and de-identify PII, and harden compute and storage with private VPCs, Secure Boot, and service perimeter controls. Reusable templates, automated jobs, and integration blueprints help teams reduce prompt injection, data leakage, and exfiltration risk while aligning operational controls with compliance and privacy expectations.

🛡️ Recent research shows the “Whisper Leak” attack can infer the topic of LLM conversations by analyzing timing and packet patterns during streaming responses. Microsoft’s study tested 30 models and thousands of prompts, finding topic-detection accuracy from 71% to 100% for some models. Providers including OpenAI, Mistral, Microsoft Azure, and xAI have added invisible padding to network packets to disrupt these timing signals. Users can further protect sensitive chats by using local models, disabling streaming output, avoiding untrusted networks, or using a trusted VPN and up-to-date anti-spyware.

🔐 The article explains how indirect prompt injection — malicious instructions embedded in external content such as documents, images, emails and webpages — can manipulate AI tools without users seeing the exploit. It contrasts indirect attacks with direct prompt injection and cites CrowdStrike's analysis of over 300,000 adversarial prompts and 150 techniques. Recommended defenses include detection, input sanitization, allowlisting, privilege separation, monitoring and user education to shrink this expanding attack surface.

🔒 Generative AI introduces new attack surfaces that alter trust relationships between users, applications and models. Siemens' pentest and security teams differentiate Offensive Security (targeted technical pentests) from Red Teaming (broader organizational simulations of real attackers). Traditional ML risks such as image or biometric misclassification remain relevant, but experts now single out prompt injection as the most serious threat — simple crafted inputs can leak system prompts, cause misinformation, or convert innocuous instructions into dangerous command injections.

✍️ Researchers from Icaro Lab (DexAI), Sapienza University of Rome, and Sant’Anna School found that short poetic prompts can reliably subvert AI safety filters, in some cases achieving 100% success. Using 20 crafted poems and the MLCommons AILuminate benchmark across 25 proprietary and open models, they prompted systems to produce hazardous instructions — from weapons-grade plutonium to steps for deploying RATs. The team observed wide variance by vendor and model family, with some smaller models surprisingly more resistant. The study concludes that stylistic prompts exploit structural alignment weaknesses across providers.

🎧 In episode 79 of The AI Fix, hosts Graham Cluley and Mark Stockley examine the latest surprises from Gemini 3, including boastful comparisons, hallucinations about the year, and reactions from industry players. They also discuss an arXiv paper proposing adversarial poetry as a universal jailbreak for LLMs and the ensuing debate over its provenance. Additional segments cover robot-versus-appliance antics, a controversial AI teddy pulled from sale after disturbing interactions with children, and whether humans need safer robots — or stricter oversight.

🚀 Amazon announced Nova 2 Omni, an all‑in‑one multimodal model in preview that accepts text, images, video, and speech inputs while producing text and image outputs. It offers a 1M token context window, supports 200+ languages for text and 10 for speech, and provides image generation/editing and multi‑speaker speech transcription with native reasoning. Early access is available to Nova Forge and authorized customers.

🚀 Amazon Bedrock expanded its model catalog with 18 new fully managed open-weight models, the largest single addition to date. The offering includes Gemma 3, Mistral Large 3, NVIDIA Nemotron Nano 2, OpenAI gpt-oss variants and other vendor models. Through a unified API, developers can evaluate, switch, and adopt these models in production without rewriting applications or changing infrastructure. Models are available in supported AWS Regions.

🎙️ Amazon announced Amazon Nova 2 Sonic, a speech-to-speech model for natural, real-time conversational AI available via Amazon Bedrock. The model delivers streaming speech understanding robust to background noise and diverse speaking styles, expressive polyglot voices, turn-taking controllability, asynchronous tool calling, and a one‑million token context window. Developers can integrate Nova 2 Sonic with Amazon Connect, leading telephony providers, open-source frameworks, and Bedrock’s bidirectional streaming API; it’s initially available in select AWS Regions.

⚠️ Researchers report that converting prompts into poetry can reliably jailbreak large language models, producing high attack-success rates across 25 proprietary and open models. The study found poetic reframing yielded average jailbreak success of 62% for hand-crafted verses and about 43% for automated meta-prompt conversions, substantially outperforming prose baselines. Authors map attacks to MLCommons and EU CoP risk taxonomies and warn this stylistic vector can evade current safety mechanisms.

⚠️ Researchers at Palo Alto Networks Unit42 found that uncensored models like WormGPT 4 and community-driven KawaiiGPT can generate functional tools for ransomware, lateral movement, and phishing. WormGPT 4 produced a PowerShell locker and a convincing ransom note, while KawaiiGPT generated scripts for credential harvesting and remote command execution. Both are accessible via subscriptions or local installs, lowering the bar for novice attackers.

🔬 Netskope Threat Labs tested whether large language models can generate operational malware by asking GPT-3.5-Turbo, GPT-4 and GPT-5 to produce Python for process injection, AV/EDR termination and virtualization detection. GPT-3.5-Turbo produced malicious code quickly, while GPT-4 initially refused but could be coaxed with role-based prompts. Generated scripts ran reliably on physical hosts, had moderate success in VMware, and performed poorly in AWS Workspaces VDI; GPT-5 raised success rates substantially but also returned safer alternatives because of stronger safeguards. Researchers conclude LLMs can create useful attack code but still struggle with reliable evasion and cloud adaptation, so full automation of malware remains infeasible today.

🤖 Amazon Lex now lets developers use Large Language Models (LLMs) as the primary natural language understanding option for voice and chat bots. Using LLMs improves handling of complex or misspelled utterances, extracts key details from verbose inputs, and enables intelligent follow‑up questions when customer intent is unclear. This capability is available in all AWS commercial regions where Amazon Connect and Amazon Lex operate, helping teams build more accurate, conversational self‑service experiences.

🛡️ Unit 42 examines the growing threat of malicious large language models that have been intentionally stripped of safety controls and repackaged for criminal use. These tools — exemplified by WormGPT and KawaiiGPT — generate persuasive phishing, credential-harvesting lures, polymorphic malware scaffolding, and end-to-end extortion workflows. Their distribution ranges from paid subscriptions and source-code sales to free GitHub deployments and Telegram promotion. The report urges stronger alignment, regulation, and defensive resilience and offers Unit 42 incident response and AI assessment services.

🚀 Anthropic's Claude Opus 4.5 is now available through Amazon Bedrock, giving Bedrock customers access to a high-performance foundation model at roughly one-third the prior cost. Opus 4.5 advances professional software engineering, agentic workflows, multilingual coding, and complex visual interpretation while supporting production-grade agent deployments. Bedrock adds two API features — tool search and tool use examples — plus a beta effort parameter to balance reasoning, tool calls, latency, and cost. The model is offered via global cross-region inference in multiple AWS regions.