What AutomationBench-AA actually tests
Artificial Analysis launched AutomationBench-AA with Zapier on July 6, 2026. It covers 657 Finance, HR, Marketing, Operations, Sales and Support workflows in 40 simulated SaaS app environments. Agents must discover APIs, update the right records and respect business rules. Programmatic checks assess the systems' final state, not just the model's persuasive answer.
Read the score without mixing denominators
At launch, Claude Fable 5 scored 48.6%, Claude Opus 4.8 reached 48.5%, Gemini 3.5 Flash 42.6% and GPT-5.5 42.1%. An important correction: AutomationBench-AA's headline score is the average share of each task's objectives achieved without a guardrail violation. It is NOT the percentage of entire tasks completed. Zapier's hosted leaderboard reports full-task completion instead, so the two percentages cannot be compared directly. Fable 5 also fell back to Opus on roughly 18% of tasks in the initial evaluation. These are launch results, not a claim about today's ranking.
What a business team should do with it
Use this test to compare execution and rule violations in a shared environment, not as a promise of success in your own workflow. Every evaluated model triggered some guardrail violations at launch, while per-task costs varied widely. Before production, trial your own flows with test data, human approvals and a check of the final CRM or app state. For a scoped pilot and a measurable path to contact, see AI for business.
Sources and methodology
Launch analysis: Artificial Analysis. Current score definition and distinction from Zapier's leaderboard: evaluation page. Zapier's hosted results: benchmarks.