All news with #llm security tag

🛡️ Microsoft released ExCyTIn-Bench, an open-source benchmarking tool to evaluate how well AI systems perform realistic cybersecurity investigations. It simulates a multistage Azure SOC using 57 Microsoft Sentinel log tables and measures multistep reasoning, tool usage, and evidence synthesis. The benchmark offers fine-grained, actionable metrics for CISOs, product owners, and researchers.

🧭 LLM-Evalkit is an open-source, lightweight application from Google that centralizes and streamlines prompt engineering using Vertex AI SDKs. It provides a no-code interface for creating, versioning, testing, and benchmarking prompts while tracking objective performance metrics. The tool promotes a dataset-driven evaluation workflow—define the task, assemble representative test cases, and score outputs against clear metrics—to replace ad-hoc iteration and subjective comparisons. Documentation and a guided console tutorial are available to help teams adopt the framework and reproduce experiments.

🔍 Amazon CloudWatch is generally available with Generative AI Observability, providing end-to-end telemetry for AI applications and AgentCore-managed agents. It expands monitoring beyond model runtime to include Built-in Tools, Gateways, Memory, and Identity, surfacing latency, token usage, errors, and performance across components. The capability integrates with orchestration frameworks like LangChain, LangGraph, and Strands Agents, and works with existing CloudWatch features and pricing for underlying telemetry.

🛡️AI developer assistants accelerate coding but introduce significant security risks across generated code, configurations, and development tools. Studies show models now compile code far more often yet still produce many OWASP- and MITRE-class vulnerabilities, and real incidents (for example Tea, Enrichlead, and the Nx compromise) highlight practical consequences. Effective defenses include automated SAST, security-aware system prompts, human code review, strict agent access controls, and developer training.

⚠️FireTail warns that Google Gemini can be tricked by hidden ASCII control characters — a technique the firm calls ASCII Smuggling — allowing covert prompts to reach the model while remaining invisible in the UI. The researchers say the flaw is especially dangerous when Gemini is given automatic access to Gmail and Google Calendar, because hidden instructions can alter appointments or instruct the agent to harvest sensitive inbox data. FireTail recommends disabling automatic email and calendar processing, constraining LLM actions, and monitoring responses while integrations are reviewed.

🔍 Daniel Miessler argues that context determines the AI attack–defense balance: whoever holds the most accurate, actionable picture of a target gains the edge. He forecasts attackers will have the advantage for roughly 3–5 years as Red teams leverage public OSINT and reconnaissance while LLMs and SPQA-style architectures mature. Once models can ingest reliable internal company context at scale, defenders should regain the upper hand by prioritizing fixes and applying mitigations faster.

🔐 Researchers uncovered a security flaw in Salesforce’s new AgentForce platform called ForcedLeak, which let attackers smuggle AI-readable instructions through a Web-to-Lead form and exfiltrate data for as little as five dollars. The hosts discuss the broader implications for AI integration, input validation, and the surprising ease of exploiting customer-facing forms. Episode 437 also critiques typical breach communications and covers ITV’s phone‑hacking drama and the Rosetta Stone story, with Graham Cluley joined by Paul Ducklin.

🔒 This AWS Security Blog post examines how Unicode tag block characters (U+E0000–U+E007F) can be abused to hide instructions inside text sent to LLMs, enabling prompt-injection and hidden-character smuggling. It explains why Java's UTF-16 surrogate handling can make one-pass sanitizers inadequate and shows recursive sanitization as a remedy, plus Python-safe filters. The post also outlines using Amazon Bedrock Guardrails denied topics or Lambda-based handlers as mitigation and notes visual/compatibility trade-offs.

⚠️ Organizations eager to deploy generative AI often underestimate the cybersecurity risks, from AI-driven phishing to model manipulation and deepfakes. The article, sponsored by Acronis, warns that many firms—especially smaller businesses—lack processes to assess AI security before deployment. It urges embedding security into development pipelines, continuous model validation, and unified defenses across endpoints, cloud and AI workloads.

🧩 Cloudflare introduces Code Mode, a new approach that converts Model Context Protocol (MCP) tool schemas into a generated TypeScript API so LLMs write code instead of emitting synthetic tool-call tokens. This lets models leverage broad exposure to real-world TypeScript, improving correctness when selecting and composing many or complex tools. Code Mode executes the generated code inside fast, sandboxed Cloudflare Workers isolates that expose only typed bindings to authorized MCP servers, preserving MCP's uniform authorization and discovery while reducing token overhead and orchestration latency.

🛡️ A Gartner survey found 29% of security leaders reported generative AI applications in their organizations were targeted by cyberattacks over the past year, and 32% said prompt-structure vulnerabilities had been deliberately exploited. Chatbot assistants are singled out as particularly vulnerable to prompt-injection and hostile prompting. Additionally, 62% of companies experienced deepfake attacks, often combined with social engineering or automated techniques. Gartner recommends strengthening core controls and applying targeted measures for each new risk category rather than pursuing radical overhauls. The survey of 302 security leaders was conducted March–May 2025 across North America, EMEA and Asia‑Pacific.

⚠️ Research and expert interviews indicate that AI coding assistants cut trivial syntax errors but increase more costly architectural and privilege-related flaws. Apiiro found AI-generated code produced fewer shallow bugs yet more misconfigurations, exposed secrets, and larger multi-file pull requests that overwhelm reviewers. Experts urge preserving human judgment, adding integrated security tooling, strict review policies, and traceability for AI outputs to avoid automating risk at scale.

🚀 VibeSDK is an open-source platform that enables organizations to deploy a complete AI-powered "vibe coding" experience with one click, integrating LLMs, secure sandboxes, and scalable hosting. It provisions isolated development environments to safely execute AI-generated code, offers templates and live previews, and automates build, test, and deploy workflows. The SDK also provides multi-model routing, observability, and caching, plus one-click export to users' Cloudflare accounts or GitHub so teams retain control of code and costs.

🛡️ SentinelOne researchers disclosed MalTerminal, a Windows binary that integrates OpenAI GPT-4 via a deprecated chat completions API to dynamically generate either ransomware or a reverse shell. The sample, presented at LABScon 2025 and accompanied by Python scripts and a defensive utility called FalconShield, appears to be an early — possibly pre-November 2023 — example of LLM-embedded malware. There is no evidence it was deployed in the wild, suggesting a proof-of-concept or red-team tool. The finding highlights operational risks as LLMs are embedded into offensive tooling and phishing chains.

⚠️A new study, “Mind the Gap,” examines time-of-check to time-of-use (TOCTOU) flaws in LLM-enabled agents and introduces TOCTOU-Bench, a 66-task benchmark. The authors demonstrate practical attacks such as malicious configuration swaps and payload injection and evaluate defenses adapted from systems security. Their mitigations—prompt rewriting, state integrity monitoring, and tool-fusing—achieve up to 25% automated detection and materially reduce the attack window and executed vulnerabilities.

🔐This article reviews emerging attack vectors against large language model assistants demonstrated in 2025, highlighting research from Black Hat and other teams. Researchers showed how prompt injections or so‑called promptware — hidden instructions embedded in calendar invites, emails, images, or audio — can coerce assistants like Gemini, Copilot, and Claude into leaking data or performing unauthorized actions. Practical mitigations include early threat modeling, role‑based access for agents, mandatory human confirmation for high‑risk operations, vendor audits, and role‑specific employee training.

🔐 Check Point has agreed to acquire Lakera, an AI-native security platform focused on protecting agentic AI and LLM-based deployments, in a deal expected to close in Q4 2025 for an undisclosed sum. Lakera’s Gandalf adversarial engine reportedly leverages over 80 million attack patterns and delivers detection rates above 98% with sub-50ms latency and low false positives. Check Point will embed Lakera into the Infinity architecture, initially integrating into CloudGuard WAF and GenAI Protect, offering near-immediate, API-based protection as an add-on for existing customers.

🔒Prisma AIRS offers unified, AI-native security across the full AI lifecycle, from model development and training to deployment and runtime monitoring. The platform focuses on five core capabilities—model scanning, posture management, AI red teaming, runtime security and agent protection—to detect and mitigate threats such as prompt injection, data poisoning and tool misuse. By consolidating workflows and sharing intelligence across Prisma, it aims to simplify operations, accelerate remediation and reduce total cost of ownership so organizations can deploy bravely.

🔒 OWASP has published an LLM AI Cybersecurity & Governance Checklist to help executives and security teams identify core risks from generative AI and large language models. The guidance categorises threats and recommends a six-step strategy covering adversarial risk, threat modeling, inventory and training. It also highlights TEVV, model and risk cards, RAG, supplier audits and AI red‑teaming to validate controls. Organisations should pair these measures with legal and regulatory reviews and clear governance.

🚀 Amazon Lex now leverages large language models to augment the natural language understanding of deterministic conversational bots in eight additional languages: Chinese, Japanese, Korean, Portuguese, Catalan, French, Italian, and German. The enhancement helps voice and chat bots parse complex utterances, tolerate spelling errors, and extract key details from verbose inputs so bots can fulfill customer requests. The capability is available in 10 commercial AWS Regions where Amazon Connect operates.