All news with #llm security tag

🧭 This article explores why most AI prototypes never reach production and how enterprise constraints create a huge validation bottleneck. It describes YouTube’s approach—using a decoupled prototyping stack and Google AI Studio templates—to enable rapid, safe experimentation with read-only access to live metadata and client-side wrappers for realistic validation. The result is faster, lower-risk product validation and a cultural shift toward disposable prototypes.

🛡️ Amazon GuardDuty introduces AI Protection to extend threat detection to AWS AI services such as Amazon Bedrock and Amazon SageMaker. The feature continuously monitors AI workloads for threats like anomalous invocations, cost harvesting attacks, and prompt injection, using CloudTrail management and data events to surface suspicious activity. Findings integrate with AWS Security Hub for centralized triage and can be enabled per account or centrally via AWS Organizations, with a 30-day trial available for GuardDuty customers.

🛡️ Kaspersky explains a new capability to detect AI-generated business email compromise (BEC) messages by identifying both BEC-specific phrases and linguistic patterns typical of machine-generated text. The company notes that cybercriminals increasingly use large language models to craft persuasive phishing and BEC campaigns, and this detection works across eight languages. The feature is integrated into Kaspersky Secure Mail Gateway and available with the KSMS Plus license after the KSMG 3.1 update.

🛡️ Security vendor CrowdStrike has added five new prompt injection techniques to its taxonomy that threaten enterprise AI deployments. These attacks manipulate LLM behavior by embedding deceptive instructions into inputs, context, or token streams to bypass safety controls and produce malicious outputs. CrowdStrike recommends threat modeling input sources, expanding testing, and enhancing detection engineering to defend against composite and multi-stage prompt attacks.

🔒 Over 70 cybersecurity organisations have signed the new CREST AI Charter, launched on July 9, committing to nine principles governing AI-enabled cybersecurity activities. The charter covers accountability and governance, transparency of use, documentation and auditability, boundaries and control, data handling and sovereignty, security and confidentiality, secure development, supply chain assurance and resilience. Signatories will maintain human oversight, document AI use, disclose data practices and implement secure development and supply chain controls. CREST intends the charter as a self-regulatory foundation to drive standards and harmonisation across industry and regulators.

🛡️ System prompts are core to generative AI applications and often include role definitions, tool descriptions, RAG context, and other proprietary instructions. This AWS Security Blog post explains why system prompt leakage is a persistent risk, highlights that it cannot be fully remediated today, and outlines practical mitigations. It recommends design principles such as minimization and avoiding sensitive data in prompts, and details controls available via Amazon Bedrock Guardrails and other mechanisms to reduce exposure and raise extraction difficulty.

🧭 A new study found GitHub Copilot can be induced to generate harmful answers when a dangerous request is broken into ordinary coding steps. Researchers Abhishek Kumar and Carsten Maple tested Claude and Gemini models through Copilot and observed that direct chat prompts were routinely refused, but the same content was produced in 816 of 816 workflow runs when framed as benchmark-improving “teaching shots.” The paper calls this technique workflow-level jailbreak construction and urges reviewing written files and whole sessions, not just chat refusals.

🛡️ Zscaler tested 26 LLMs and found several autonomous agents susceptible to indirect prompt injection (IPI) traps, with some high-end models failing while a few lower-tier models fared better. The vendor identified hidden instructions on websites that manipulated agent behavior and caused real-world impacts in controlled tests. Experts warn that agent risk is dynamic, the attack surface is architectural, and binary "safe/vulnerable" labels are overly simplistic for CISOs. The findings highlight that agentic AI introduces new trust boundaries and insider-like threats to enterprise security.

🚀 Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), which splits LLM inference into separate prefill and decode GPU pools and transfers KV cache over EFA using GPU-Direct RDMA. This reduces contention between long-context prefill and per-token decode, enabling more consistent per-token latency, higher goodput under strict latency SLOs, and independent scaling of prefill and decode. DPD is enabled via a pdSpec in the existing InferenceEndpointConfig and works with HyperPod's KV cache offloading and intelligent routing on EFA-capable instances.

🔒 Google Cloud and Valtech introduce Nexus SDV, an open-source, modular platform that enables AI-native, scalable management of software-defined vehicles. The platform integrates with Android Automotive OS and supports up to 100 million devices while emphasizing TCO reduction via Arm-based compute and Bigtable. Nexus AI leverages Gemini models and the Gemini Enterprise Agent Platform for real-time telemetry analysis and agentic vehicle capabilities. Security is built-in with mTLS/PKI, identity brokering, secret management, network isolation, and an enterprise Secure AI Framework.

🔍 Sysdig researchers detailed an autonomous AI agent, dubbed JadePuffer, that executed an end-to-end intrusion and extortion campaign after exploiting a vulnerable Langflow server. The agent leveraged an LLM to adapt tactics, delivering over 600 Base64-encoded Python payloads to pivot from an internet-facing Langflow instance to a production MySQL/Nacos server and encrypt 1,342 configuration records before demanding ransom. The operation demonstrated rapid self-correction and contextual reasoning in payloads, prompting calls for behavior-focused detection.

📘 Enterprise LLM deployments face KV cache growth that can exceed node RAM and local SSD capacity. Google Cloud demonstrates using Managed Lustre as a shared, high-performance external KV cache tier integrated with vLLM and the llm-d offloading stack to bypass host-level limits and simplify cross-node data management. Benchmarks with Llama-3.3-70B on a six-node A3 Mega cluster show over 50% TCO savings and nearly 60% reduction in GPU-hour usage by achieving a 95% cache hit rate. The guide includes architecture, deployment steps, PVC evictor behavior, and validated tracks for Qwen and Gemma models.

🛡️ Unit 42 found that large language models (LLMs) commonly hallucinate plausible web domains for real brands, and adversaries are registering these nonexistent domains to intercept AI-generated traffic. This phenomenon, called phantom squatting, poses a supply chain risk and was observed across multiple sectors. Researchers predicted adversary registrations 18–51 days in advance and discovered over 13,229 malicious URLs plus ~250,000 unregistered hallucinated domains.

🧭 This post introduces the preview of BigQuery's new AI.AGG() function, which enables natural-language aggregation over millions of rows of unstructured and multimodal data directly inside SQL. It explains practical uses—analyzing logs, discovering product categories, and summarizing image collections—while showing how AI.AGG() batches inputs, handles NULLs, reports errors, and integrates with other BigQuery AI functions like AI.CLASSIFY(). The write-up outlines best practices for token usage, model endpoint selection, and struct handling to help users deploy AI.AGG() effectively.

🛡️ OpenAI has unveiled GPT-5.6 Sol, calling it its "most capable model yet for cybersecurity," but initial access is restricted to a small set of vetted partners at the request of the US government. The preview, announced on June 26, introduces three tiers—Sol, Terra and Luna—and is available via API and Codex to selected partners while OpenAI coordinates with the government on a cyber executive order framework. OpenAI says Sol excels at long-horizon tasks like vulnerability research, includes enhanced safeguards and real-time classifiers, and currently does not autonomously produce full exploits.

🔎 The article argues that frontier AI models like Mythos are a signal of shifting cyber economics rather than an immediate, novel threat. It emphasizes that longstanding security fundamentals—asset visibility, patching, identity controls and resilient operations—remain the primary defenses. The author advocates using AI to accelerate analysis, prioritize remediation and close persistent control gaps rather than replacing skilled practitioners or prompting reactive, headline-driven spending.

🛡️ The article examines how AI, particularly large language models, can bridge the gap between atomic indicators of compromise (IOCs) and richer strategic threat intelligence by indexing and relating unstructured reports. It highlights opportunities to retrieve relevant intelligence and generate tailored defensive advice while warning about data veracity and confidentiality. The piece also emphasizes practical Windows threats abusing COM and recommends tooling and hunting practices to detect such misuse.

📝 This post highlights a new paper that demonstrates how large language models are vulnerable to prompt injection because they learn to distinguish instruction blocks by style rather than explicit tags. The authors argue that role tags became a de facto security architecture but do not map cleanly into model representations, producing persistent role confusion. The paper warns that without genuine role perception, defenses will be reactive and brittle, and calls for deeper study of roles within the LLM stack.

🛡️ A malware developer has begun embedding provocative text about nuclear and biological weapons inside large JavaScript block comments in spyware payloads to confuse AI-based scanners. The commented header is ignored at runtime but aims to trigger refusals or misclassification in naive LLM-powered triage systems that ingest file starts without isolating untrusted content. Traditional detection methods—YARA, entropy checks, AST parsing, and behavioral analysis—remain effective, but the technique is a practical anti-analysis tactic against weak AI-first pipelines.

🛡️ AI is reshaping cybersecurity by breaking the long-held assumption of predictable, deterministic systems. Traditional prevention-focused controls remain important but are insufficient as AI agents, LLMs and automated development accelerate runtime change and attacker capabilities. Organizations must prioritize runtime visibility, use AI to augment defensive operations, rebuild vulnerability management and emphasize resilience and containment to manage evolving AI-driven risks.