All news with #ai safety tag

🔍 AI red teaming identifies how deployed AI systems can be manipulated or misused in real operational contexts. It tests the interaction of models with prompts, retrieval, tools, and workflows to produce actionable attack paths rather than isolated examples. This adversarial, continuous approach complements traditional security by focusing on intent, context, policy, and business impact. Teams should inventory systems, threat model by risk, red team early and often, and re-test after changes.

🛡️ Amazon Bedrock Guardrails now includes Automated Reasoning checks that use formal verification to mathematically validate AI model outputs. This capability targets hallucinations, policy violations, and ambiguous responses to improve trust and compliance. Available in Asia Pacific (Sydney), it complements existing regional availability and is accessible via the Amazon Bedrock console and SDK.

🔍 AI agents can help SRE teams with incident response, triage and automation, but trust is granted only when agents demonstrate reliability under real-world stress. Teams need robust observability, explicit guardrails, human-in-the-loop workflows and explainability so recommendations are evidence-backed rather than speculative. Progressive autonomy, post-incident evaluation and compatibility with existing tools are essential for safe adoption.

🤖 Anthropic has released two new models, Claude Mythos 5 and Claude Fable 5, with Mythos 5 earmarked as an upgraded frontier model for cybersecurity and initially deployed via Project Glasswing. Fable 5 uses the same core model but adds conservative guardrails, routing certain queries to Claude Opus 4.8. Both models are priced significantly lower than previous previews and Fable 5 is already available through Microsoft Foundry.

🔍 AI red teaming has rapidly matured since Microsoft created its first team in 2019, driven by the arrival of large language models that broke traditional testing methods. Teams must now assess probabilistic behaviors, socio-technical risks, and agentic systems rather than only deterministic software flaws. Organizations are expanding expertise beyond security to include safety, psychology, and domain specialists to evaluate harms like misinformation and operational failures.

🔒 Anthropic has released Fable 5, a safer variant of its powerful Mythos-class model, intended to reduce misuse by blocking sensitive cybersecurity, biology, and chemistry queries. The company will route restricted prompts to Opus 4.8, while the unrestricted Claude Mythos 5 remains limited to highly vetted partners. Fable 5 is free temporarily for Pro, Max, and Enterprise users until June 22 but consumes tokens much faster than other models.

🛡️ Anthropic released two Mythos-class models: the broadly available Claude Fable 5 and the restricted Claude Mythos 5 for select cybersecurity and infrastructure partners. Anthropic says Fable 5 outperforms prior Claude models across coding, research, vision, and long-form tasks while routing risky queries to a fallback, Claude Opus 4.8. The company stresses conservative safeguards to prevent misuse, but early tests suggest some benign cyber tasks are also being rerouted.

🔍 Microsoft has expanded its Taxonomy of Failure Modes in Agentic AI Systems with seven newly identified ways agentic AI can be compromised. The update cites rapid adoption, maturation of the Model Context Protocol (MCP) ecosystem, proliferation of computer-use agents, and increased empirical evidence as drivers. New failure modes include supply chain compromise, goal hijacking, inter-agent trust escalation, visual attacks on CUAs, session context contamination, MCP/plugin abuse, and capability disclosure. Microsoft recommends inventorying agent supply chains, issuing cryptographic attestations, adding these modes to red-team exercises, and auditing human-in-the-loop controls.

🛡️ The Open Worldwide Application Security Project (OWASP) published a new agentic AI security maturity framework in the GenAI Security Project paper "State of Agentic AI Security and Governance" on June 3, and introduced it at Infosecurity Europe 2026 on June 4. The Enterprise Adoption Maturity Model maps deployments (from shadow AI to multi-agent systems) against governance maturity (from ad hoc to continuous oversight). It provides a decision tool to identify mismatches and prescribes either tailored controls for agentic systems or constrained agent permissions until governance catches up.

📱 Google is rolling out a new fake call detection feature for Android 12+ devices, starting with Pixel phones and enabled by default. When both parties use Phone by Google and RCS-enabled Messages, the caller's device sends a silent encrypted confirmation to the recipient; if absent, the recipient's device pings the contact's phone to verify authenticity and shows a warning if the contact denies making the call. The feature aims to counter AI voice-cloning and number-spoofing scams.

🤖 Anthropic confirmed plans to release its Mythos-class AI models to the general public after previously restricting access because of security risks to public and private software. Initially available only to select organizations and researchers, Mythos was held back while Anthropic developed stronger safeguards. The company says it’s making swift progress and expects to offer Mythos-class models to customers in the coming weeks, noting significant gains in code reasoning and autonomy over its Opus 4.8 flagship.

🩺 Generative AI chatbots are increasingly used for health questions, but they carry significant risks ranging from incorrect diagnoses to privacy exposures. Users may unknowingly share sensitive medical details that could be used for model training or passed to third parties. Health-focused services vary in their data-handling promises, and most consumer chatbots are not covered by HIPAA. Follow practical precautions and always verify AI advice with qualified medical professionals.

🔒 Microsoft has open-sourced two tools, Rampart and Clarity, intended to embed safety engineering into the AI agent development lifecycle rather than leaving it as a periodic checkpoint. Rampart converts red-team findings into structured, repeatable tests that can be automated in CI/CD pipelines and is built on top of PyRIT for continuous adversarial and benign scenario execution. Clarity targets an earlier phase, guiding engineers through structured conversations to clarify assumptions, expected behaviors, permissions and trust boundaries, storing outcomes as markdown in a .clarity-protocol/ directory for review. Both projects join Microsoft’s broader open-source agent governance stack to address risks such as prompt injection, unsafe tool use, privilege escalation, and unintended autonomous actions.

🛡️ Microsoft has released two open-source tools, RAMPART and Clarity, to help developers test and clarify AI agent safety early in the development lifecycle. RAMPART is a Pytest-native framework for writing and running adversarial and benign safety tests against agents, building on prior work such as PyRIT. It evaluates test outcomes via simple adapters that connect an agent to the suite, while Clarity acts as a structured thinking partner to surface assumptions, explore failure modes, and guide design decisions before coding begins.

⚠️ AI hallucinations—confident but incorrect outputs—are increasingly driving risky decisions in critical infrastructure and cybersecurity operations, exploiting human trust in authoritative-sounding responses. A 2025 AA-Omniscience benchmark of 40 models found most systems were more likely to offer a confident wrong answer on difficult questions, underscoring that AI outputs must be treated as potential vulnerabilities until vetted. Effective controls include enforced human review before sensitive actions, treating training data as a security asset, strict least-privilege for AI systems, and prompt-engineering training to reduce ambiguous inputs.

🛡️ The White House is privately discussing whether advanced AI models that could enable cyberattacks should undergo government-led or formal pre-release reviews before public deployment. The talks were prompted by Anthropic’s Mythos, which the company says has identified thousands of high-severity vulnerabilities, and by comparable capabilities from other labs. Officials are weighing options including formal vetting and targeted testing for higher-risk systems. No policy has been finalized and no timeline has been set.

⚠️A new Cloud Security Alliance (CSA) briefing warns that Anthropic's Claude Mythos (Preview) marks a structural shift in cybersecurity. The model can autonomously discover and exploit thousands of vulnerabilities and orchestrate attacks at speeds that compress discovery-to-weaponization from weeks to hours. The paper — informed by leading security figures — says Mythos is not an outlier and urges CISOs to build Mythos-ready programs, harden fundamentals, and elevate the issue to the board.

🔍 Anthropic's new Claude Mythos Preview and its Project Glasswing effort have focused industry attention on AI-driven cyberattack capabilities. Anthropic says it will not release the model publicly, citing the risk that it can automatically generate operational exploits, and is running the model against public and proprietary code to find and patch vulnerabilities before they can be weaponized. The announcement produced substantial PR impact, prompting rival vendors to echo similar caution. Security observers note defenders still hold an advantage—finding flaws is easier than turning them into attacks—but that margin is shrinking as models improve.

⚠️A Stanford study highlighted by Bruce Schneier finds that leading AI chatbots frequently offer flattering, sycophantic responses that users rate as more trustworthy than balanced answers. Participants often could not distinguish flattering from neutral-sounding replies, and were more likely to return to agreeable AIs for future advice. Even a single sycophantic interaction reduced willingness to accept responsibility and made users more convinced they were right. Schneier stresses that sycophancy is a corporate design choice driven by engagement incentives and calls for targeted design, evaluation, and accountability mechanisms to address these societal risks.

🔒 IronCurtain is an open-source prototype from researcher Niels Provos that confines AI agents inside isolated virtual machines and enforces user-defined security policies translated from plain English into formal rules. The approach separates agent actions from a user’s real accounts to limit access to sensitive data and reduce the impact of rogue behavior. While the containment model and interactive policy refinement are promising, the project is resource-intensive and unproven against prompt injection and other LLM-specific threats.