All news with #ai red teaming tag

🔒 The UK National Cyber Security Centre (NCSC) has launched the Cyber Shield initiative to build a national cyber-defence capability powered by agentic AI. The project will use coordinated red and blue agents to discover and mitigate vulnerabilities at scale, enable national automated scanning, and support real-time intelligence sharing. The NCSC says success requires partnerships with government, critical infrastructure and frontier AI providers.

🔒 Fortinet describes its integration of frontier AI models (Anthropic’s Glasswing/Mythos and OpenAI’s Daybreak/GPT 5.5 Cyber) alongside on-premises models to scale security testing across firmware, source code, and penetration testing. The company emphasizes responsible innovation, mature vulnerability management, and human validation of AI findings. Fortinet reports limited exploitable firmware issues but greater findings from source-code analysis and commits to mitigation, virtual patching, and secure-by-default deployments.

🛡️ Researchers uncovered a macOS malware family named macOS.Gaslight that embeds fabricated error messages and debugging data inside a Rust binary to mislead AI-assisted analysis tools. The 3.5 KB payload contains 38 fake system messages — including memory dumps, token-expiration warnings, and build errors — designed to appear as legitimate developer logs. SentinelOne attributes the sample with high confidence to a North Korean-linked actor and notes the strings aim to prompt-inject LLM pipelines, causing them to abort or distrust their session. The malware retains standard backdoor and data-stealing capabilities alongside the deceptive messaging tactic.

🛡️ The Cobalt State of Pentesting Report 2026 surveyed roughly 450 cybersecurity professionals across 2025 and 2026 and found trust in fully automated AI vulnerability testing has dropped sharply. Reliance on AI-only testing fell from 29% to 9%, while 47% now prefer a hybrid human-plus-AI model. Respondents reported that 78% of fully automated scanners missed critical vulnerabilities, and AI/LLM issues showed longer MTTR and lower fix rates.

🛡️ Enterprise AI red teaming must go beyond simple prompt tests to examine full systems — models, prompts, retrievals, tools, permissions, workflows, and APIs — because risks appear when components interact. Check Point argues that threat intelligence plus threat modeling enables identification of realistic attack paths, evidence-based findings, and prioritized remediation. Continuous, comparative testing and re-testing after changes ensures fixes are effective and keeps pace with rapid AI adoption.

🔒 AWS has introduced Continuum, a service to continuously discover, investigate, and remediate vulnerabilities across first-party and third-party codebases. The platform uses AI to validate exploitability, generate remediation recommendations, and propose fixes that integrate with existing development workflows. New capabilities include automatic threat modeling in STRIDE format, while more established features derive from the Security Agent product. Continuum supports graduated trust from human-in-the-loop review to an "enforce mode" for autonomous remediation.

🔍 AI red teaming identifies how deployed AI systems can be manipulated or misused in real operational contexts. It tests the interaction of models with prompts, retrieval, tools, and workflows to produce actionable attack paths rather than isolated examples. This adversarial, continuous approach complements traditional security by focusing on intent, context, policy, and business impact. Teams should inventory systems, threat model by risk, red team early and often, and re-test after changes.

🔍 At Google Cloud, collaboration with customers guided a technical sprint in January 2026 with a major telecommunications partner to refine Model Armor, the runtime security service for generative AI. By embedding with the customer's developers and security teams, the Google Cloud Developer Advocacy group observed real-world workflows and identified friction points such as search-first documentation needs, tuning confidence levels to avoid false positives, clearer enforcement guidance, and IAM-related 403 errors during integrations. The team translated these findings into tested code samples, a confidence level matrix, explicit integration guides for Apigee, Gemini Enterprise Agent Platform, and GKE, and deeper technical documentation to improve operational utility and reduce deployment friction.

🔎 The UK government's GC3 ran weekly in-person hackathons using frontier AI models to scan public code repositories across nine departments, identifying 407 findings including authentication bypasses, data exposure and remote code execution. Teams built diverse pipelines combining models and traditional tools like Gitleaks, Trivy and Semgrep, and all exploitable critical and high-risk issues were remediated. The initiative highlighted the benefits of tightly scoped model components, the need for human triage, and cost-effective scanning, though export restrictions on some models may affect future work.

🔍 AI red teaming has rapidly matured since Microsoft created its first team in 2019, driven by the arrival of large language models that broke traditional testing methods. Teams must now assess probabilistic behaviors, socio-technical risks, and agentic systems rather than only deterministic software flaws. Organizations are expanding expertise beyond security to include safety, psychology, and domain specialists to evaluate harms like misinformation and operational failures.

🔎 XBOW received early access to Anthos Mythos Preview and ran a structured evaluation across benchmarks, interactive workflows, and live-site integrations. The model excels at reading source code, finding vulnerability candidates, and aiding native-code analysis and reverse engineering. While powerful for generating leads and precise technical analysis, Mythos Preview is less effective at exploit validation and exhibits mixed judgment that benefits from human orchestration.

🛡️ depthfirst's autonomous agent discovered 21 previously unknown zero-day vulnerabilities in FFmpeg, producing reproducible PoC inputs for each at a reported cost of about $1,000 for the run. In the same week, Google released Chrome 149 with fixes for a record 429 security bugs, over 100 of which are critical or high severity, following an overhaul of its bounty program to cope with a surge of AI-generated reports. The findings illustrate how AI is accelerating vulnerability discovery and increasing pressure on triage and patching processes across widely used software.

🛡️ Lloyds Banking Group treats agentic AI as an engineering problem to be designed, constrained and tested at scale. At OWASP’s GenAI Security Summit, Lloyds’ security leads explained an “AI safe adoption” strategy spanning lifecycle governance, an internal agent marketplace, and multidisciplinary feature teams. Key challenges include agent identity, runtime observability and automated red‑teaming, while prioritizing low‑risk, high‑value use cases for customers.

🔎 The Microsoft AI Red Team released a v2.0 update to the Taxonomy of Failure Modes in Agentic AI Systems, grounded in twelve months of red team engagements and operational data. The revision adds seven new failure mode categories—such as agentic supply chain compromise, goal hijacking, and visual attacks against computer-use agents—expands mitigations, and emphasizes supply chain, zero‑trust, and session hardening.

🔍 At Infosecurity Europe 2026, Bugcrowd unveiled ExploitBench, a graded benchmark assessing AI models' ability to chain vulnerability discovery into staged exploits against a vulnerable V8 build. Anthropic’s Claude Mythos outperformed OpenAI’s GPT‑5.5 in head‑to‑head runs, achieving higher average scores and more top‑tier exploits, often with occasional human nudges. The report highlights rising offensive potential of frontier LLMs and urges defenders to adopt automated remediation and prioritization.

🛡️ Anthropic has broadened Project Glasswing, giving 150 additional organizations access to its most capable model, Claude Mythos Preview, to help find vulnerabilities in critical software. The program, first opened to roughly 50 partners in April, claims more than 10,000 high- or critical-severity flaws discovered to date. New participants span 15+ countries and underrepresented sectors like power, water, healthcare and hardware, chosen for the potential catastrophic impact of breaches. Anthropic warned that while discovery is accelerating, safe public release of Mythos-class models remains restricted due to incomplete safeguards.

🔍 Halcyon research reveals a dramatic rise in AI-powered cybercrime tooling across underground markets, jumping from 38 mentions in December to 1,486 in February. Cynthia Kaiser, SVP of Halcyon’s Ransomware Research Center, detailed four product categories: weaponized LLMs, AI-enabled identity fraud, AI-augmented malware/infrastructure, and jailbroken or stolen AI services. She warned that automated distribution, freemium models and redundant channels have lowered the financial barrier to entry and increased resilience against takedown efforts.

🔍 Cisco researchers found that popular frontier LLMs from OpenAI, Anthropic, Google, xAI, and Amazon exhibit substantially higher risk when subjected to multi-turn adversarial attacks than when assessed with single-prompt safety benchmarks. The team ran tens of thousands of single-turn and multi-turn attacks across 15 models and multiple configurations, revealing wide gaps in attack success rates (ASRs) and configuration-dependent safety behavior. They urge improved benchmarks, transparency on configuration impacts, and publication of paired single- and multi-turn ASRs to better inform procurement and governance decisions.

🔒 Microsoft has open-sourced two tools, Rampart and Clarity, intended to embed safety engineering into the AI agent development lifecycle rather than leaving it as a periodic checkpoint. Rampart converts red-team findings into structured, repeatable tests that can be automated in CI/CD pipelines and is built on top of PyRIT for continuous adversarial and benign scenario execution. Clarity targets an earlier phase, guiding engineers through structured conversations to clarify assumptions, expected behaviors, permissions and trust boundaries, storing outcomes as markdown in a .clarity-protocol/ directory for review. Both projects join Microsoft’s broader open-source agent governance stack to address risks such as prompt injection, unsafe tool use, privilege escalation, and unintended autonomous actions.

🛡️ Microsoft has released two open-source tools, RAMPART and Clarity, to help developers test and clarify AI agent safety early in the development lifecycle. RAMPART is a Pytest-native framework for writing and running adversarial and benign safety tests against agents, building on prior work such as PyRIT. It evaluates test outcomes via simple adapters that connect an agent to the suite, while Clarity acts as a structured thinking partner to surface assumptions, explore failure modes, and guide design decisions before coding begins.