All news with #ai red teaming tag

🛡️ OpenAI revealed GPT‑Red, an internal automated red‑teaming model that simulates human adversaries to discover prompt injection vulnerabilities at scale. GPT‑Red iteratively probes and refines attacks against production models, helping harden GPT‑5.6 Sol and reduce prompt injection failures by 6× versus GPT‑5.5. OpenAI uses self‑play reinforcement learning to train both attacker and defender models while keeping GPT‑Red segregated to avoid misuse.

🔍 AI security cannot be treated as a one-time certification; it requires an ongoing cycle of adversarial discovery, hardening, and operational resilience. NIST research shows no finite set of guardrails can guarantee permanent robustness, so teams must continuously test, remediate, and monitor systems as models, prompts, and integrations evolve. Effective programs tie red teaming to runtime protection and governance so findings become durable improvements.

🔎 This article from Google Data Cloud explores a rigorous, information‑theoretic approach to evaluating AI data agents by converting binary pass/fail tests into graded difficulty sweeps. The team introduces Discovery Bench and iterative surprisal-based query refinement (iSQR) to generate low/medium/high ambiguity variations of queries, quantify surprisal, and map where agents succeed or fail. The piece highlights how this method reveals cliffs and sweet spots in agent behavior and calls for auditing benchmarks themselves to avoid misleading conclusions.

🛡️ Security vendor CrowdStrike has added five new prompt injection techniques to its taxonomy that threaten enterprise AI deployments. These attacks manipulate LLM behavior by embedding deceptive instructions into inputs, context, or token streams to bypass safety controls and produce malicious outputs. CrowdStrike recommends threat modeling input sources, expanding testing, and enhancing detection engineering to defend against composite and multi-stage prompt attacks.

🔒 The UK National Cyber Security Centre (NCSC) has launched the Cyber Shield initiative to build a national cyber-defence capability powered by agentic AI. The project will use coordinated red and blue agents to discover and mitigate vulnerabilities at scale, enable national automated scanning, and support real-time intelligence sharing. The NCSC says success requires partnerships with government, critical infrastructure and frontier AI providers.

🔒 Fortinet describes its integration of frontier AI models (Anthropic’s Glasswing/Mythos and OpenAI’s Daybreak/GPT 5.5 Cyber) alongside on-premises models to scale security testing across firmware, source code, and penetration testing. The company emphasizes responsible innovation, mature vulnerability management, and human validation of AI findings. Fortinet reports limited exploitable firmware issues but greater findings from source-code analysis and commits to mitigation, virtual patching, and secure-by-default deployments.

🛡️ Researchers uncovered a macOS malware family named macOS.Gaslight that embeds fabricated error messages and debugging data inside a Rust binary to mislead AI-assisted analysis tools. The 3.5 KB payload contains 38 fake system messages — including memory dumps, token-expiration warnings, and build errors — designed to appear as legitimate developer logs. SentinelOne attributes the sample with high confidence to a North Korean-linked actor and notes the strings aim to prompt-inject LLM pipelines, causing them to abort or distrust their session. The malware retains standard backdoor and data-stealing capabilities alongside the deceptive messaging tactic.

🛡️ The Cobalt State of Pentesting Report 2026 surveyed roughly 450 cybersecurity professionals across 2025 and 2026 and found trust in fully automated AI vulnerability testing has dropped sharply. Reliance on AI-only testing fell from 29% to 9%, while 47% now prefer a hybrid human-plus-AI model. Respondents reported that 78% of fully automated scanners missed critical vulnerabilities, and AI/LLM issues showed longer MTTR and lower fix rates.

🛡️ Enterprise AI red teaming must go beyond simple prompt tests to examine full systems — models, prompts, retrievals, tools, permissions, workflows, and APIs — because risks appear when components interact. Check Point argues that threat intelligence plus threat modeling enables identification of realistic attack paths, evidence-based findings, and prioritized remediation. Continuous, comparative testing and re-testing after changes ensures fixes are effective and keeps pace with rapid AI adoption.

🔒 AWS has introduced Continuum, a service to continuously discover, investigate, and remediate vulnerabilities across first-party and third-party codebases. The platform uses AI to validate exploitability, generate remediation recommendations, and propose fixes that integrate with existing development workflows. New capabilities include automatic threat modeling in STRIDE format, while more established features derive from the Security Agent product. Continuum supports graduated trust from human-in-the-loop review to an "enforce mode" for autonomous remediation.

🔍 AI red teaming identifies how deployed AI systems can be manipulated or misused in real operational contexts. It tests the interaction of models with prompts, retrieval, tools, and workflows to produce actionable attack paths rather than isolated examples. This adversarial, continuous approach complements traditional security by focusing on intent, context, policy, and business impact. Teams should inventory systems, threat model by risk, red team early and often, and re-test after changes.

🔍 At Google Cloud, collaboration with customers guided a technical sprint in January 2026 with a major telecommunications partner to refine Model Armor, the runtime security service for generative AI. By embedding with the customer's developers and security teams, the Google Cloud Developer Advocacy group observed real-world workflows and identified friction points such as search-first documentation needs, tuning confidence levels to avoid false positives, clearer enforcement guidance, and IAM-related 403 errors during integrations. The team translated these findings into tested code samples, a confidence level matrix, explicit integration guides for Apigee, Gemini Enterprise Agent Platform, and GKE, and deeper technical documentation to improve operational utility and reduce deployment friction.

🔎 The UK government's GC3 ran weekly in-person hackathons using frontier AI models to scan public code repositories across nine departments, identifying 407 findings including authentication bypasses, data exposure and remote code execution. Teams built diverse pipelines combining models and traditional tools like Gitleaks, Trivy and Semgrep, and all exploitable critical and high-risk issues were remediated. The initiative highlighted the benefits of tightly scoped model components, the need for human triage, and cost-effective scanning, though export restrictions on some models may affect future work.

🔍 AI red teaming has rapidly matured since Microsoft created its first team in 2019, driven by the arrival of large language models that broke traditional testing methods. Teams must now assess probabilistic behaviors, socio-technical risks, and agentic systems rather than only deterministic software flaws. Organizations are expanding expertise beyond security to include safety, psychology, and domain specialists to evaluate harms like misinformation and operational failures.

🔎 XBOW received early access to Anthos Mythos Preview and ran a structured evaluation across benchmarks, interactive workflows, and live-site integrations. The model excels at reading source code, finding vulnerability candidates, and aiding native-code analysis and reverse engineering. While powerful for generating leads and precise technical analysis, Mythos Preview is less effective at exploit validation and exhibits mixed judgment that benefits from human orchestration.

🛡️ depthfirst's autonomous agent discovered 21 previously unknown zero-day vulnerabilities in FFmpeg, producing reproducible PoC inputs for each at a reported cost of about $1,000 for the run. In the same week, Google released Chrome 149 with fixes for a record 429 security bugs, over 100 of which are critical or high severity, following an overhaul of its bounty program to cope with a surge of AI-generated reports. The findings illustrate how AI is accelerating vulnerability discovery and increasing pressure on triage and patching processes across widely used software.

🛡️ Lloyds Banking Group treats agentic AI as an engineering problem to be designed, constrained and tested at scale. At OWASP’s GenAI Security Summit, Lloyds’ security leads explained an “AI safe adoption” strategy spanning lifecycle governance, an internal agent marketplace, and multidisciplinary feature teams. Key challenges include agent identity, runtime observability and automated red‑teaming, while prioritizing low‑risk, high‑value use cases for customers.

🔎 The Microsoft AI Red Team released a v2.0 update to the Taxonomy of Failure Modes in Agentic AI Systems, grounded in twelve months of red team engagements and operational data. The revision adds seven new failure mode categories—such as agentic supply chain compromise, goal hijacking, and visual attacks against computer-use agents—expands mitigations, and emphasizes supply chain, zero‑trust, and session hardening.

🔍 At Infosecurity Europe 2026, Bugcrowd unveiled ExploitBench, a graded benchmark assessing AI models' ability to chain vulnerability discovery into staged exploits against a vulnerable V8 build. Anthropic’s Claude Mythos outperformed OpenAI’s GPT‑5.5 in head‑to‑head runs, achieving higher average scores and more top‑tier exploits, often with occasional human nudges. The report highlights rising offensive potential of frontier LLMs and urges defenders to adopt automated remediation and prioritization.

🛡️ Anthropic has broadened Project Glasswing, giving 150 additional organizations access to its most capable model, Claude Mythos Preview, to help find vulnerabilities in critical software. The program, first opened to roughly 50 partners in April, claims more than 10,000 high- or critical-severity flaws discovered to date. New participants span 15+ countries and underrepresented sectors like power, water, healthcare and hardware, chosen for the potential catastrophic impact of breaches. Anthropic warned that while discovery is accelerating, safe public release of Mythos-class models remains restricted due to incomplete safeguards.