Zapier launches independent ranking to measure whether AI actually automates business tasks

What AutomationBench-AA is

Artificial Analysis, working with Zapier, launched AutomationBench-AA on July 6, 2026: an independent leaderboard built to answer a practical question. Can an AI agent automate real business workflows without breaking the rules a company sets for its own operations? Unlike generic academic benchmarks, which usually measure abstract reasoning or isolated problem solving far removed from daily work, AutomationBench-AA simulates the daily routine of Finance, HR, Marketing, Operations, Sales and Support teams, checking whether an agent completes a full task while respecting the guardrails any company applies to its own operations. It is a benchmark built for application, not theory.

The numbers: 657 tasks, 40 apps and the results

The methodology relies on 657 tasks spread across those six business areas, run inside 40 simulated replicas of popular SaaS tools such as Gmail, Slack, Salesforce, Zendesk and HubSpot. Those replicas reproduce how the real platforms behave, which allows testing an agent under conditions close to an actual workplace without exposing real customer data. The first ranking put Claude Fable 5 in the lead, completing 48.6% of tasks without violating business rules, closely followed by Claude Opus 4.8 at 48.5%. Next came Gemini 3.5 Flash at 42.6% and GPT-5.5 at 42.1%. The data is published on Zapier's official benchmarks page, in the project's public GitHub repository and in Artificial Analysis's announcement article.

Why it matters for everyday AI users

For anyone deciding which AI to plug into marketing, support or operations automation, AutomationBench-AA offers something rare: a practical selection criterion, built from tasks that resemble what those teams actually do, instead of theoretical tests far removed from daily routines. Knowing that one model completes nearly half of its tasks without breaking a business rule, while another trails by several percentage points, is direct, usable information for anyone deciding where to place automation with less operational risk. In a marketing team, for instance, that difference can mean an agent that launches a campaign while respecting the company's approval steps versus one that skips a step and creates rework.

What the numbers still hide

Even the leading models, Claude Fable 5 and Claude Opus 4.8, stay below 50% completion without rule violations. That shows AI driven business task automation is still far from reliable enough to run without human oversight, no matter how advanced the underlying model is. AutomationBench-AA, maintained by Artificial Analysis and Zapier, should work as an ongoing gauge of that progress as new models join the ranking and the gap between lab performance and production performance becomes clearer over time, giving teams a steadily updated reference instead of a one time snapshot.