Balancing Model Tradeoffs for AI in SOCs
🔎 Cisco Talos evaluated 66 model-and-reasoning combinations from Anthropic and OpenAI on a tool-assisted log-review task to determine practical tradeoffs for SOC and DFIR workflows. Reviewers used common Unix tools to decide if a synthetic dataset was real, and every condition was scored by four persona-based reviewers across multiple panels. Results measured investigative quality, cost, time, and consistency, revealing that the highest accuracy models were often slower, costlier, and sometimes unreliable due to refusals or format failures. Talos recommends using a Pareto-frontier approach and benchmarking each reasoning level and persona for operational selection.
