All news with #llm security tag

🧠 Researchers at Carnegie Mellon University demonstrated that leading large language models (LLMs), by themselves, struggle to execute complex, multi-host cyber-attacks end-to-end, frequently wandering off-task or returning incorrect parameters. Their proposed solution, Incalmo, is a structured abstraction layer that constrains planning to a precise set of actions and validated parameters, substantially improving completion and coordination. The work highlights both enhanced offensive potential when LLMs are scaffolded and urgent defensive challenges for security teams.

🛠️ Martin Lee recounts a weekend experiment using an AI agent to assist with a personal software project. The model provided valuable architectural guidance, flawless boilerplate, and resolved a tricky threading issue, delivering a clear productivity lift. However, generated code failed to match real library APIs, used incorrect parameters and fictional functions, and lacked sufficient input validation. After manual debugging Lee produced a working but not security-hardened prototype, highlighting remaining risks.

🛡️The Phishing Triage Agent is now in Public Preview and automates triage of user-reported suspicious emails within Microsoft Defender. Using large language models, it evaluates message semantics, inspects URLs and attachments, and detects intent to classify submissions—typically within 15 minutes—automatically resolving the bulk of false positives. Analysts receive natural‑language explanations and a visual decision map for each verdict, can provide plain‑language feedback to refine behavior, and retain control via role‑based access and least‑privilege configuration.

🔒 Microsoft outlines a layered defense-in-depth strategy to protect systems using LLMs from indirect prompt injection attacks. The approach pairs preventative controls such as hardened system prompts and Spotlighting (delimiting, datamarking, encoding) to isolate untrusted inputs with detection via Microsoft Prompt Shields, surfaced through Azure AI Content Safety and integrated with Defender for Cloud. Impact mitigation uses deterministic controls — fine-grained permissions, Microsoft Purview sensitivity labels, DLP policies, explicit user consent workflows, and blocking known exfiltration techniques — while ongoing research (TaskTracker, LLMail-Inject, FIDES) advances new design patterns and assurances.