An AI agent benchmark is a standardized test designed to evaluate the performance of AI agents on various tasks. These tests are intended for comparing different agents' capabilities in areas such as programming, reasoning, performing tasks on a websites, and so on. They have significant overlap with language model benchmarks. See the overview section of that article for some foundational concepts not repeated here. They also have significant conceptual overlap with benchmarks in reinforcement learning and robotics, because the very concept of an "AI agent" is formulated from reinforcement learning theory, and the most prominent examples of AI agents before the early 2020s were robots.
Overview AI agent benchmarks generally grew out of language model benchmarks in the early 2020s, and has significant overlap with benchmarks designed for reasoning language models. This gradual growth is particularly clear in the case of a lineage of benchmarks for question-answering. Originally, the task is a purely reactive language task: given a trivia question, answer it. It was saturated, and thus made more difficult: given a trivia question and a reference document, answer it. It was saturated again, and thus made more difficult: given a trivia question, use a web-browser to find the reference document to answer it, resulting in BrowseComp. This shows how a type of benchmarks for purely input-output language models evolved into a type of benchmarks for AI agents.
Certain benchmarks are specialized for evaluating LLM-based AI agents. Compared to standard benchmarks, such benchmarks are significantly more more difficult to construct and run.
Construction Agency benchmarks significantly more costly to construct.
Unlike standard benchmarks, in an agency benchmark, a trial runs in a loop of action-observation. The agent takes in an observation and produces an action, while an environment takes in an action and produces an observation. Thus, unlike all other benchmarks, agency benchmarks cannot be specified by static data (such as a textual description of the task), but require an interactive environment, which is specified programmatically. The environment usually consists of several parts. Common components of the environment include a code repository that the agent should work on, utility programs the agent might use, and a runtime environment for running the entire trial. Each task is often containerized to allow exact replicability.
It is difficult to programmatically evaluate performance on many agentic tasks. The agent might solve the task in an unintended way, or fail the evaluator despite actually solving it in an intended way. Consequently, the trial rollouts are often inspected for phenomena such as reward-hacking behavior or broken evaluator.
Running Agency benchmarks are also significantly more costly to run.
As short-horizon tasks saturate, more difficult benchmarks are constructed that test for longer-horizon tasks. This means the agent-environment interaction takes longer. This has increased the time-per-trial from several minutes to several hours.
At the beginning, language model would take a query and produce an answer immediately. With the invention of chain-of-thought reasoning in language models, a language model can be prompted repeatedly for many tokens before producing a final answer. This means the cost per answer is variable. Thus, a higher cost tends to produce an answer more likely to be correct, and thus a higher benchmark score. This has two effects:
When running a benchmark on an agent, the task cost itself is recorded, and agents are compared for not just their benchmark performance, but on a cost-performance curve. The same agent is benchmarked multiple times at different task cost levels. This makes running a benchmark more costly.
There are several types of "cost" usually used: the wallclock time, the output's token count, and the monetary cost.
The two effects are exacerbated with the advent of agent harnesses, since the same underlying model can result in very different agents, when plugged into different agent harnesses.
As frontier model capability increases, agency benchmarks are increasingly subject to unique problems not found in other types of benchmarks.
In certain areas, frontier agents have saturated essentially all benchmarks. This is particularly true of mathematics, where frontier progress is now measured by solving open problems. Frontier agents increasingly reward-hack, acting to increase scores in a way that violates the task requirement. This is similar to Goodhart's law, but done by the agents rather than the researchers. Frontier agents are increasingly evaluation-aware, i.e. aware that they are acting in an agency benchmark. This is similar to observer effect. In recent instances, such as the OpenAI–HuggingFace incident, agents escaped the benchmark sandbox.
List of benchmarks
General AgentBench: 1014 test tasks and 269 development tasks in 8 interactive environments (operating system, database, web browsing, games, etc). Each tasks require multiple rounds (estimated 5–50) of interaction. Agents' Last Exam: long-horizon, professional computer tasks, currently containing 1500+ tasks collected toward a 5000-task target. ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence): Given three pairs of before-and-after diagrams of applying a rule, apply the same rule to the fourth before-diagram. It is similar to a Raven's Progressive Matrices test. Later versions explicitly test for agentic behavior in a game-like interactive environment. HCAST (Human-Calibrated Autonomy Software Tasks): 189 tasks in machine learning, cybersecurity, software engineering, and general reasoning. Each task has a "baseline", the measured average time required for a human skilled in the task domains, working under identical conditions as AI agents. The baseline ranges from 1 minute to 8+ hours. Produced by METR in 2025-03.
Tool use Some agentic benchmarks are not particularly difficult in themselves, and test instead for proficiency in using tools that an AI agent would use, such as web browsing, using a command line interface, calling web APIs, etc. BrowseComp: 1,266 questions that requires internet browsing for finding a short factual answer. Adversarial against GPT-4o with and without browsing, OpenAI o1, and an early version of the Deep Research model. GAIA: 450 questions with unambiguous answers that require information that can be obtained by browsing the Internet, requiring different levels of tooling and autonomy to solve. Divided into 3 difficulty levels. Mind2Web: 2,350 tasks collected from 137 websites, and crowdsourced action sequences. The task is to reproduce the action sequence. SpreadsheetBench: 912 real-world spreadsheet manipulation tasks scraped from public Excel help forums, spanning formula writing, data cleaning, filtering and layout edits in diverse formatting. Scored automatically on 2729 test cases at cell-, sheet- and overall levels. WebArena: 241 mock-up websites based on real-world websites (Reddit, GitLab, Magento's admin portal, etc), and 812 tasks to be performed on the websites. The tasks include information-seeking, site navigation, and content and configuration operation. WebVoyager: 643 multimodal tasks based on 15 popular websites. Evaluation is by screenshotting the action sequence and asking a vision language model to judge. Terminal-bench: A collection of 66 complex tasks in the Linux terminal. Updated in versions. OSWorld: 369 multimodal computer-using tasks, involving multiple real web and desktop apps and OS file I/O. In both Windows and Ubuntu. Each task includes an initial state setup configuration, and is tested by an execution-based evaluation script. Updated to OSWorld-Verified in 2025-07. Windows Agent Arena: 154 multimodal tasks with the same format as OSWorld. Only in Windows.
Economic Some benchmarks are specifically designed to measure capability of AI agents to perform economically productive tasks that are currently performed by humans. These tasks are usually obtained by taking particular tasks in a job, and specifying them accurately enough so that they can be verified automatically. These benchmarks are usually developed in collaboration with companies in the industry that currently use humans to perform these tasks. AutomationBench: a benchmark developed by Zapier, on cross-application workflow orchestration via REST APIs, using 47 real tools across 6 business functions (Sales, Marketing, Operations, Support, Finance, and HR). Based on real monthly tasks executed by Zapier itself. APEX-agents: a benchmark showing how leading AI models hold up doing actual white-collar work tasks, such as consulting, investment banking, and law. Vals Index, run by Vals. Aggregation of benchmarks across 4 sectors (Finance, Coding, Legal, and Tax) weighted in proportion to its share of U.S. GDP. Intended to measure the potential economic impact. Harvey's Legal Agent Benchmark: 1200 tasks for legal work, across 24 practice areas, evaluated by over 75000 expert-written rubric. The task is to produce legal work (such as legal discovery) based on instruction and a folder of legal documents. Part of the Vals Index GDPval: 1,320 tasks evaluating AI agents on real-world knowledge-work deliverables across 44 occupations from the top 9 U.S. GDP-contributing industries. Tasks made by professionals averaging 14 years of experience. The full evaluation is by blinded expert pairwise comparison against human expert deliverables, each comparison costing 1 human hour on average. A gold subset of 220 tasks is publicly available. The public set can be graded by a finetuned version of GPT-5. The held-out set of 1,100 tasks is only available internally in OpenAI, and external models cannot be submitted for grading on this set. TAU-bench (Tool-Agent-User benchmark, also written as τ-bench): Two environments (retail, airline booking) that test for an agent to fulfill user instructions, interactively over multiple turns of dialogue. The user is simulated by a language model. Updated to TAU2-bench (τ²-bench), which focuses on telecom applications. Tasks are synthesized by LLM-generated product requirements document, agent database schema, agent tools, and user environments ("mocked phone"). Updated to TAU3-bench (τ³-bench), which adds tests for operate over large collections of internal company documents, and duplex voice.
Programming
ScienceAgentBench: 102 multimodal data science tasks, each being a real scientific data-driven discovery problem reframed as a code-generation task. Agents must produce a complete Python program file that implements the task, can run in isolation, and saves its outputs. In the domains of Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience. Sourced from 44 peer-reviewed publications that have released their code and data under permissive licenses. Each task is validated by domain experts. SWE-bench: 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase and an issue, the task is to edit the codebase to solve the issue. It has 2 subsets: Lite (300 problems that are faster to run), Verified (human-validated subset of 500 problems reviewed by software engineers). The Verified version is deprecated by OpenAI in 2026-02, since many remaining failures reflected flawed or underspecified tests. They recommended the public split of SWE-bench Pro. SWE-bench Pro was released later, containing 1,865 harder items. This has 3 splits: public, held-out, and proprietary (only available to a select set of early-stage startups). Multi-SWE-bench: 1,632 problems across 7 languages: Java, TypeScript, JavaScript, Go, Rust, C, and C++. Similar to SWE-bench.