1 hour ago
1
Benchmarking AI models on coding puzzles and trivia is easy. Benchmarking them on whether they can actually do a job is much harder. Artificial Analysis just took a serious run at the latter.
The independent AI research and evaluation firm has launched EnterpriseOps-Gym-AA, a platform designed to measure how well large language model-based agents complete real, multi-step tasks inside live enterprise systems. Think filing an IT ticket, resolving a customer service issue, or navigating HR workflows, end-to-end, with no hand-holding.
The early results are telling. Claude Fable 5, Anthropic’s latest model, leads the inaugural leaderboard with a 51.1% task success rate under oracle tool mode, a configuration that includes adaptive reasoning and fallback mechanisms. That number sounds modest until you see where the field was sitting before: the original ServiceNow paper that inspired this style of evaluation reported a top score of 37.4% for models including Claude Opus 4.5, during evaluations conducted in March 2026.
Going from 37.4% to 51.1% in the span of a few months is meaningful progress. It is also a reminder that even the best model in the world still fails nearly half the time on tasks a competent office worker handles before lunch.
What the benchmark actually tests
EnterpriseOps-Gym-AA is not a multiple-choice quiz. It runs 1,150 curated tasks spread across eight enterprise domains, including Customer Service, HR, and IT Service Management.
The environment itself is the interesting part. Tasks are run inside a containerized sandbox built around 164 database tables and 512 functional tools. In plain terms, agents are dropped into something that behaves like a real enterprise software stack and told to get things done.
Scoring is deliberately strict. Performance is graded on the final state of the database after the agent finishes, with no partial credit awarded. Either the task is done correctly, or it is not.
To keep results reliable, Artificial Analysis runs each task three times using the Stirrup agent harness. Averaging across repetitions smooths out variance and makes the scores more trustworthy than a single-run evaluation would be.
Artificial Analysis is operating these evaluations independently. The firm has built a track record of running comparable leaderboards, including AutomationBench-AA and Harvey LAB-AA, positioning itself as a neutral scorekeeper in a space where vendors have obvious incentives to grade their own homework.
Why this benchmark matters for the enterprise AI market
For enterprise buyers evaluating AI agent deployments, the 51.1% ceiling on the current best model is important context. It means that even under optimal conditions, with oracle tool access and fallback mechanisms enabled, the leading model is still failing on roughly half of structured enterprise tasks.
Claude Fable 5 was released by Anthropic on June 9, 2026. Its performance here represents the current high-water mark for this class of evaluation, though the benchmark is designed to be continuously updated as new models emerge. The jump from Claude Opus 4.5’s 37.4% to Claude Fable 5’s 51.1% suggests that capability improvements are real and accelerating, which is exactly the kind of signal enterprise technology buyers and investors in the AI sector are watching closely.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.
English (US) · 