< ciso
brief />
Tag Banner

All news with #model poisoning tag

21 articles

Fake Video Calls and AI Sleeper Agents on Smashing Security

🎙️ Smashing Security episode 493 explores a sneaky scam in which victims join a fake video call and are pushed to install a malicious microphone update. The FBI says the attackers replay genuine footage of earlier victims to build trust. The episode also covers the Lazarus exploit from Microsoft Research, which shows how AI models can be turned into sleeper agents, and how NordLayer AI Scout helps tackle shadow AI.
read more →

Researchers Hijack Hugging Face Account of Kokoro AI Model

🔐 Researchers took control of the publisher account for Kokoro, a popular text-to-speech model on Hugging Face, by registering an expired domain and resetting the password. They then showed that altering only the model's weights could change its spoken output, with no malicious code for scanners to detect. The case highlights a broader supply-chain trust problem across the open-model ecosystem, as the platform does not verify ongoing account ownership.
read more →

CrowdStrike Details Detection of Backdoored LoRA Models

🛡️ CrowdStrike researchers have outlined a two-stage method for detecting backdoored LoRA models before deployment. Open-weight models are now a core part of the software supply chain, yet organizations have no easy way to verify that a downloaded model was not tampered with. The approach combines static weight analysis with trigger recovery to turn vague suspicion into an actionable finding for analysts.
read more →

Post‑Training Supply Chain Backdoors in Coding Agents

🛡️ Two Microsoft teams demonstrate how post-training supply-chain attacks can implant latent backdoors in coding and terminal agents. Dataset poisoning hides crafted examples in fine-tuning data to teach agents to identify and misuse available command-execution tools, achieving ~78% success in-distribution. Model poisoning assumes control of training and distributed checkpoints, combining SFT and RL to reach 84–95% success on unseen repositories. Both attacks exploit trust in datasets and model weights and highlight the need for data scrutiny, checkpoint provenance, runtime evaluation, and strict tool permissions.
read more →

Tiny neuron edits can create sleeper backdoors in LLMs

🔒 Microsoft researchers demonstrated that rewriting fewer than 10 dormant neurons in an open-weight model can implant a sleeper backdoor that activates on a simple trigger, with no retraining required. The method, called Lazarus, alters neurons in the final feed-forward layer to recognize triggers and inject attacker-controlled commands into outputs, enabling agents built on the model to execute those commands. Microsoft recommends auditing model weights, verifying provenance, and restricting agent permissions to mitigate the risk.
read more →

Researchers Demonstrate Neuron-Level Backdoor Method

🔍 Microsoft researchers show a technique to implant a hidden backdoor by editing fewer than 10 rarely used neurons in an AI model without full retraining. The backdoor triggers when a specific string appears, causing the model to output attacker-specified commands that an agent could execute with its own permissions. The team tested the method, named Lazarus, across multiple open-weight models and published mitigation guidance to help teams detect and limit such risks.
read more →

NemoClaw vulnerability lets local AI be poisoned

🛡️ A vulnerability in Nvidia's NemoClaw can let a malicious website trick a browser into reaching a locally running Ollama model server via DNS rebinding, giving unauthenticated API access. Cyera researchers showed an attacker could modify a model's chat template to inject persistent, hidden instructions that survive future sessions. Nvidia has patched macOS and Linux builds in NemoClaw 0.0.35, while Windows/WSL remains unpatched.
read more →

Google Vertex AI SDK bucket squatting enables RCE

🔒 A design flaw in the Vertex AI SDK for Python allowed attackers to hijack model staging buckets across projects by predicting bucket names derived from project ID and region. Unit 42 researchers called this class of issue Bucket Squatting, where global bucket name uniqueness enabled pre-creation and silent takeover. The flaw could lead to cross-tenant model poisoning and remote code execution via pickle deserialization. Google issued fixes in SDK versions 1.144.0 and 1.148.0 and users should upgrade.
read more →

Vertex AI SDK bucket-squatting enables RCE

🛡️ We discovered a vulnerability in the Google Cloud Vertex AI Python SDK that allowed an attacker to hijack a model upload and poison it, enabling remote code execution (RCE) in a victim's serving infrastructure. The issue stems from a predictable default staging bucket name and a missing ownership check in the SDK. By creating the same deterministic bucket in their own project and granting broad permissions, an attacker could replace uploaded model artifacts within a short window before Vertex AI reads them. Google fixed the issue in google-cloud-aiplatform v1.148.0 released April 15, 2026; developers should upgrade to the patched SDK.
read more →

Democratization of AI and the Rising Data Poisoning Threat

⚠️ Recent research shows that as few as 250 fabricated documents or images can measurably alter large language model behavior, making data poisoning accessible to non-experts. Online communities and influencers are already seeding false content that may be ingested during public-model training or fine-tuning. Organizations should maintain a clean 'gold' model, monitor input streams for anomalous patterns, and perform regular adversarial testing to detect drift and backdoors before deployment.
read more →

Microsoft Builds Scanner to Detect Backdoors in LLMs

🔍 Microsoft has developed a lightweight scanner to detect backdoors in open-weight large language models (LLMs) by evaluating three observable signals tied to internal model behavior. The tool extracts memorized content, isolates suspect substrings, and scores candidates with loss functions that formalize attention and output anomalies. The approach requires no additional training and runs across common GPT‑style models, but it needs access to model files and is best suited for trigger-based, deterministic backdoors.
read more →

Detecting Backdoored Language Models at Scale — Practical Scanner

🔍 Microsoft researchers released new findings and a practical scanner for detecting backdoors in open-weight language models. The study identifies three signatures — a distinctive “double triangle” attention pattern, leakage of poisoning training data through memorization, and trigger “fuzziness” — and uses them to reconstruct likely triggers without retraining. The scanner requires only forward passes, works on GPT-like models, and was validated across 270M–14B models and common fine-tuning regimes. The team notes limits: it needs model file access, favors deterministic backdoors, and should be used as part of layered defenses.
read more →

The AI Fix #84: Hungry ghost, data poisoning, Grok

🤖 In episode 84 of The AI Fix, hosts Graham Cluley and Mark Stockley survey a series of recent AI developments that raise practical and philosophical questions. They discuss reports that Grok will be integrated into Pentagon networks, a campaign by insiders to poison training data, and research showing small amounts of tainted data can sway model behavior. The episode also covers Google removing AI health overviews after risky outputs, findings that asking a model the same question twice can improve answers, and surprising advances in automated theorem solving.
read more →

Weird Generalizations and Inductive Backdoors in LLMs

⚠️ Recent research demonstrates that small amounts of narrow finetuning can produce broad, unexpected shifts in LLM behavior. The authors show weird generalization—models adopting outdated worldviews from bird-naming examples—and introduce inductive backdoors, where models learn triggers and behaviors via generalization. These effects enable persona hijacking and hard-to-detect misalignment.
read more →

Top Cyber Threats Targeting AI Systems and Infrastructure

🔒 AI systems face a growing range of attacks—from data poisoning and model poisoning during training to adversarial inputs, prompt injection, and model theft during deployment. These threats exploit weak data governance, supply chain dependencies, and inadequate monitoring. Security leaders should adopt proactive controls including provenance tracking, adversarial testing, rate limits, and routine red teaming. Frameworks like MITRE ATLAS can help map attacker techniques and prioritize defenses.
read more →

Picklescan Flaws Enable Malicious PyTorch Model Execution

⚠️ Picklescan, a Python pickle scanner, has three critical flaws that can be abused to execute arbitrary code when loading untrusted PyTorch models. Discovered by JFrog researchers, the issues — a file-extension bypass (CVE-2025-10155), a ZIP CRC bypass (CVE-2025-10156) and an unsafe-globals bypass (CVE-2025-10157) — let attackers present malicious models as safe. The vulnerabilities were responsibly disclosed on June 29, 2025 and fixed in Picklescan 0.0.31 on September 9; users should upgrade and review model-loading practices and downstream automation that accepts third-party models.
read more →

CIO’s First Principles: A Reference Guide to Securing AI

🔐 Enterprises must redesign security as AI moves from experimentation to production, and CIOs need a prevention-first, unified approach. This guide reframes Confidentiality, Integrity and Availability for AI, stressing rigorous access controls, end-to-end data lineage, adversarial testing and a defensible supply chain to prevent poisoning, prompt injection and model hijacking. Palo Alto Networks advocates embedding security across MLOps, real-time visibility of models and agents, and executive accountability to eliminate shadow AI and ensure resilient, auditable AI deployments.
read more →

Digital Health Needs Security at Its Core to Scale AI

🔒 The article argues that AI-driven digital health initiatives proved essential during COVID-19 but simultaneously exposed critical cybersecurity gaps that threaten pandemic preparedness. It warns that expansive data ecosystems, IoT devices and cloud pipelines multiply attack surfaces and that subtle AI-specific threats — including data poisoning, model inversion and adversarial inputs — can undermine public-health decisions. The author urges security by design, including zero-trust architectures, data provenance, encryption, model governance and cross-disciplinary drills so AI can deliver trustworthy, resilient public health systems.
read more →

The AI Fix #73: Gemini gambling, poisoning LLMs and fallout

🧠 In episode 73 of The AI Fix, hosts Graham Cluley and Mark Stockley explore a sweep of recent AI developments, from the rise of AI-generated content to high-profile figures relying on chatbots. They discuss research suggesting Google Gemini exhibits behaviours resembling pathological gambling and report on a Gemma-style model uncovering a potential cancer therapy pathway. The show also highlights legal and security concerns— including a lawyer criticised for repeated AI use, generals consulting chatbots, and techniques for poisoning LLMs with only a few malicious samples.
read more →

Agentic AI and the OODA Loop: The Integrity Problem

🛡️ Bruce Schneier and Barath Raghavan argue that agentic AIs run repeated OODA loops—Observe, Orient, Decide, Act—over web-scale, adversarial inputs, and that current architectures lack the integrity controls to handle untrusted observations. They show how prompt injection, dataset poisoning, stateful cache contamination, and tool-call vectors (e.g., MCP) let attackers embed malicious control into ordinary inputs. The essay warns that fixing hallucinations is insufficient: we need architectural integrity—semantic verification, privilege separation, and new trust boundaries—rather than surface patches.
read more →