An automated safety testing framework for LLM agents has revealed severe vulnerabilities across four production agent systems, finding that multi-channel attacks succeed an average of 93.9 percent of the time.
The framework, called Vera, instantiates software engineering testing principles for non-deterministic AI agents through a three-stage, self-reinforcing pipeline. It was evaluated on four production agent frameworks, OpenClaw, Hermes, Codex, and Claude Code, and the accompanying Vera-Bench benchmark includes 1,600 executable safety cases spanning 124 risk categories across three execution settings.
Vera’s approach differs from existing safety testing in three key ways. First, it uses literature-driven exploration to continuously discover and structure emerging risks into taxonomies of safety hazards, attack methods, and tool execution environments, rather than relying on a fixed, expert-designed list of violations. Second, it generates safety cases combinatorially across taxonomy dimensions, producing concrete initial states and deterministic verification predicates grounded in observable artifacts. Third, it runs agents in isolated sandboxes where a control agent steers multi-turn interaction based on runtime observations, and evidence-grounded verifiers judge outcomes from environment state and tool-call evidence rather than model self-report.
The 93.9 percent average attack success rate under multi-channel attacks, where an adversary combines multiple attack vectors simultaneously, underscores a fundamental weakness in current safety architectures. Single-vector attacks were less effective, suggesting that existing guardrails are tuned for known threat models but break down under combinatorial pressure.
Vera’s modular, infrastructure-focused design addresses a growing concern in the AI safety community: as agentic systems evolve rapidly, hard-coded safety rules and manual red-teaming cannot keep pace. By treating safety testing as an automated, extensible pipeline, similar to continuous integration in software engineering, Vera provides a path toward rigorous and maintainable safety evaluation for rapidly evolving systems.
The code and benchmark are publicly available.
Sources: Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification (arXiv, July 2, 2026); Vera codebase