TECHNOLOGY EXPLAINER
Coding Agents and Computer Use
— from code generation to autonomous development and screen operation: what they can do, and the pitfalls of measuring them
AI writes code, runs tests and fixes failures, and operates a mouse and keyboard while looking at a browser or a screen. The range of what can be handed over is widening fast, but evaluating it is hard, and benchmark values move a great deal with the quality of problems, the weakness of tests and leakage into training data. This article sets out, from primary sources, the stages from code generation to autonomous software development, evaluations such as SWE-bench and their limits, how agents that look at a screen and operate it work and where they struggle, and how to think about permissions and sandboxes.
- What coding agents and computer use are (the short version)
- From code generation to autonomous development
- How coding agents work
- Evaluation and its limits — centered on SWE-bench
- Agents that operate browsers and screens
- Permissions and sandboxes
- Limits, and what could not be confirmed
- How to read evaluations and standards
- Glossary
- References (primary sources)
- Claim-to-source audit
Sourced = stated in published material (link given)
Our calculation = a figure this article derived, with the assumptions spelled out
Not yet confirmed = a plan, schedule or draft whose outcome has not been confirmed
Other readings are marked separately as Commentary. Inference is shown in bold.
This field moves extremely fast, and the benchmarks themselves are rebuilt within months. The dates and numbers here are what could be confirmed in public documents on 15 October 2026, and benchmark numbers are values for the models and settings at the time of each paper or report. They are not the performance of current models, and numbers from different settings or evaluators are not lined up to rank anything. Numbers a developer gives about its own product are treated as "the developer's statement", and this article does not rank particular products or companies. Concrete procedures of misuse are not written.
1. What coding agents and computer use are (the short version)
- A coding agent loops through "read code, edit, run tests, look at the result": the unit of work has widened from code completion of a single function to fixing a task across several files until the tests pass. Evaluation has moved from function-level benchmarks to real repositories' issues and long work at a terminalSourced [Sources 1, 8, 16]
- A screen-operation agent loops through "look at a screenshot, click coordinates, look at the screen again": it can operate apps that have no API, but it is slow, mislocates coordinates, and risks following instructions inside the screen. Where a January 2025 report gave OSWorld 38.1% (humans 72.4%), OSWorld 2.0 in 2026 shows even the best completing only about a fifth of long tasksSourced [Sources 27, 21]
- Benchmark values move a great deal with the quality of problems and tests: solution leakage and weak tests were found in SWE-bench, and OpenAI stopped reporting Verified in February 2026. Keep permissions minimal, run the agent inside a sandbox, and have a person check irreversible operationsSourced [Sources 11, 10, 30]
"The AI's code passed the tests" is not the same as "it is correct", and "it felt faster" is not the same as "it was faster". In METR's controlled experiment, experienced developers forecast before starting that AI would cut completion time by 24% and estimated afterwards that it had cut it by 20%, yet measured completion time was 19% longer on tasks where AI was allowed (a result for early-2025 tools and conditions)Sourced [Source 5].
2. From code generation to autonomous development
"AI writes code" can be sorted into stages by the unit of work (this article's grouping). The higher the stage, the wider the scope a person must check and the impact of failure.
(1) Completing a function: HumanEval and pass@k
Codex (Chen et al., 2021) is a GPT-family model fine-tuned on publicly available code from GitHub, and a production version is said to power GitHub Copilot. On HumanEval, which measures the functional correctness of programs synthesized from docstrings, Codex solves 28.8% of the problems (GPT-3 0%, GPT-J 11.4%), and with "repeated sampling" of 100 samples per problem it solves 70.2%. The paper also lists difficulty with docstrings describing long chains of operations and with binding operations to variablesSourced [Source 1]. MBPP (Austin et al.) has 974 tasks designed to be solvable by entry-level programmers; the paper reports that the largest model solves 59.6% with few-shot prompting alone and that natural-language feedback from a human halves the error rate [Source 2]. On simulated Codeforces competitions with more than 5,000 participants, AlphaCode ranked in the top 54.3% on average, and the paper says large-scale sampling and filtering by program behavior were key [Source 3]Sourced.
Our calculation Codex's pass@1 on HumanEval is 28.8%. If the success probability of one attempt were 0.288 on every problem and attempts were independent, the probability of at least one success in 100 attempts would be 1 - (1 - 0.288)100 = 1 - 0.712100 (subtracting about 1.8 x 10-15), that is almost 100%. Yet the paper's solve rate with 100 samples is 70.2%.
The gap shows that success rates differ greatly between problems (easy problems are solved every time, hard ones never, however often one tries). Applying the average rate p to every problem does not hold. Assumptions and limits: this is a simple calculation of the consequence of assuming independence and an identical success rate; the per-problem distribution of Codex's success rates was not checked beyond the paper.
(2) Generation and editing in dialogue: productivity experiments
In the controlled experiment by Peng et al., the group using an AI pair programmer (GitHub Copilot) completed the task of implementing an HTTP server in JavaScript 55.8% faster than the group without itSourced [Source 4]. This was a simple, well-defined task, though, and another experiment discussed in Section 4 showed a different result.
(3) Resolving whole repositories' issues: SWE-bench and designs for agents
SWE-bench (Jimenez et al., 2023) has 2,294 problems made from real GitHub issues and the pull requests that fixed them in 12 popular Python repositories. Given a codebase and an issue description, the model edits the code to resolve the issue. Resolving them often needs understanding and coordinating changes across several functions, classes and files, and at the time of the paper the best model, Claude 2, solved only 1.96%Sourced [Source 8].
SWE-agent (Yang et al.) argues that a custom agent-computer interface (ACI) designed for agents greatly improves the ability to create and edit files, navigate repositories and run tests, and reports pass@1 of 12.5% on SWE-bench and 87.7% on HumanEvalFix (state of the art then) [Source 17]. Agentless (Xia et al.), by contrast, lets the LLM decide no next actions and uses just a simple three-phase process of localization, repair and patch validation; it reports 32.00% (96 correct fixes) on SWE-bench Lite, the highest among open-source agents then, at a cost of $0.70, asking whether complex autonomous agents are really needed [Source 18]. OpenHands is an open, MIT-licensed platform for developing agents that, like a human developer, write code, work at a command line and browse the web [Source 19]Sourced.
3. How coding agents work
The basis of a coding agent is the LLM + tools + a loop covered in this series' explainer on how AI agents work. The tools are reading and writing files, search, and running commands (including tests).
View original SVG, values and source annotations
- The loop: read and search code, edit, run tests or commands, look at the result (failure log), and repeat until the tests pass or a maximum number of turns is reached (conceptual figure). The SWE-agent paper says that the design of the tools (the ACI) then governs performance [Source 17]
- Permissions: according to the Claude Code documentation, manual mode starts with read-only permissions and asks before editing files or running commands, whether to approve once or allow from then on. The same document cautions that the working-directory boundary is a "permission prompt", so a Bash command you approve can still write anywhere your user account canSourced [Source 31]
- Sandbox: Anthropic says that asking for confirmation every time causes "approval fatigue", in which people stop looking closely at what they approve, and describes a method that lets the agent work freely inside defined filesystem and network boundaries and notifies the user when it tries to reach outside them (the company says permission prompts fell by 84% in its internal use)Sourced [Source 30]. Section 6 covers this in detail
- Long work at a terminal: Terminal-Bench (Merrill et al., 2026) is Terminal-Bench 2.0, 89 hard tasks in computer terminal environments inspired by real workflows, each with a unique environment, a human-written solution and comprehensive tests for verification. The paper reports that frontier models and agents score below 65%Sourced [Source 16]
4. Evaluation and its limits — centered on SWE-bench
(1) The course of SWE-bench, and the flaws pointed out
SWE-bench became a "standard" for evaluation, while problems of quality were pointed out one after another. Here they are in chronological orderSourced.
| When (source) | What happened (as the source describes it) | Reading |
|---|---|---|
| October 2023 SWE-bench [Source 8] | 2,294 problems. The best at the time of the paper was Claude 2 at 1.96% | At first a set of tasks that "almost nothing solves" |
| 13 August 2024 SWE-bench Verified (OpenAI) [Source 9] | 93 Python-experienced developers manually screened 1,699 samples. 68.3% were removed for under-specification (38.3% flagged), unit tests that rule out valid solutions (61.1% flagged) and other issues (with a note that this is likely overzealous). The 500 retained are Verified. GPT-4o reached 33.2% with the best scaffold, more than double its 16% on the original SWE-bench. OpenAI states that the original dataset underestimated abilities | Fixing problem quality raised the value |
| October 2024 SWE-Bench+ (Aleithan et al.) [Source 11] | Examining by hand the issues solved by SWE-Agent + GPT-4, then at the top, the authors report that in 32.67% of successful patches the solution was given in the issue text or comments (solution leakage) and that 31.08% were suspicious because of weak tests. Without these, the resolution rate fell from 12.47% to 3.97%. Lite and Verified have the same problems, and over 94% of issues were created before the LLMs' knowledge cutoff dates | Fixing problem quality lowered the value (the portion that had leaked) |
| June 2025 UTBoost (Yu et al., ACL 2025) [Source 12] | Augmenting tests automatically, the authors identified 36 task instances with insufficient tests and 345 erroneous patches wrongly labeled as passed in the original SWE-bench. This affected 40.9% of SWE-Bench Lite and 24.4% of Verified leaderboard entries, with 18 and 11 ranking changes respectively | Weak tests change even the ranking |
| September 2025 SWE-Bench Pro (Deng et al.) [Source 13] | 1,865 problems from 41 actively maintained repositories (public 11, held-out 12, commercial 18). Includes long tasks that may take hours to days, human-verified; described as contamination-resistant | An attempt to be harder and harder to contaminate |
| 23 February 2026 OpenAI's report [Source 10] | Verified is "increasingly contaminated". Auditing a 27.6% subset that models often failed to solve, OpenAI found at least 59.4% of the audited problems had flawed tests that reject functionally correct submissions. All frontier models it tested could reproduce the human-written gold patch or verbatim problem-statement specifics for some tasks, indicating they had seen problems and solutions in training. It also notes state-of-the-art progress on Verified slowing from 74.9% to 80.9% in six months. OpenAI stopped reporting Verified scores, recommends other developers do so too, and for now recommends reporting SWE-bench Pro | "Improvement" increasingly reflects exposure to the benchmark more than the model's ability |
All Sourced (descriptions in each paper or report). The two OpenAI pages block automated retrieval, so they were checked in Internet Archive copies.
Our calculation From the values in the sources, we calculate the size of the changes.
- SWE-Bench+: 12.47% to 3.97%. The fall is (12.47 - 3.97) / 12.47 = about 68%; the ratio of new to old is 3.97 / 12.47 = about 0.32
- SWE-bench Verified (OpenAI): 16% to 33.2%. The ratio is 33.2 / 16 = about 2.1 times
- The mismatch in the experiment (METR): the developers' post-hoc estimate was "20% shorter", the measurement "19% longer". The gap between estimate and measurement is about 39 points (20 + 19)
Assumptions and limits: the three examples differ in model, scaffold, subject and date and are not for comparison with one another. They illustrate that even under the same name "SWE-bench", the meaning of a value changes with the version (original, Lite, Verified, Pro) and the evaluation settings.
(2) Other evaluations
LiveCodeBench (Jain et al.) collects new problems over time from contests on LeetCode, AtCoder and CodeForces to avoid leakage into training data. It measures not only code generation but also self-repair, code execution and test-output prediction, and argues that existing benchmarks such as HumanEval and MBPP may be overfit [Source 14]. SWE-Lancer (Miserendino et al.) uses more than 1,400 freelance tasks from Upwork (worth $1 million in real payouts) and reports that frontier models still cannot solve the majority of them [Source 15]Sourced.
(3) Productivity and security: other sides of code "quality"
METR's controlled experiment (Becker et al., 2025) had 16 experienced open-source developers complete 246 tasks in mature projects where they had an average of five years of experience, randomly assigning each task to "AI allowed" or "AI disallowed", and reported that allowing AI increased completion time by 19%. Developers had forecast a 24% reduction and, afterwards, estimated 20%, and the predictions of economics and machine-learning experts (39% and 38% shorter) were also the opposite. The authors say the influence of experimental artifacts cannot be entirely ruled out, but that the robustness of the effect makes it unlikely to be primarily a function of the experimental designSourced [Source 5].
On 24 February 2026, METR announced that it was changing the design of the experiment. It judged that the data from the new experiment begun in August 2025 (10 of the original developers plus 47 new ones) gives an unreliable signal of the current productivity effect, mainly because of selection effects such as developers declining to take part because they do not wish to work without AI. The raw results are a speedup of 18% for the subset of original developers (confidence interval from 38% faster to 9% slower) and 4% for newly recruited developers (15% faster to 9% slower); both intervals straddle zero. The company says it believes developers are likely more sped up in early 2026 than in early 2025, but that its data is only very weak evidence for the size of that increaseSourced [Source 6]. So the primary sources support this reading: the effect of AI coding assistance on productivity was a lengthening in early 2025, and in 2026 it is "not measured reliably (a speedup is likely, its size unknown)".
On security, Pearce et al. had Copilot generate code in 89 scenarios involving high-risk CWEs and reported that about 40% of the 1,689 programs were vulnerable (a result for the 2021 tool, not for current tools)Sourced [Source 7].
The lesson of SWE-bench is that weak tests let wrong changes "pass" (SWE-Bench+, UTBoost) [Sources 11, 12]. The same applies when asking AI to write a script for analyzing measurement data: "the tests passed" means something only if the tests are strict enough. In materials and process work, prepare as "expected-value tests", before having AI write the code, measured values of known reference samples, representative values worked out by hand, unit and order-of-magnitude checks, and boundary conditions. A general framework for test design is in ISO/IEC/IEEE 29119-1:2022, and an overview of testing AI systems in ISO/IEC TS 42119-2:2025 (neither text was checked, as they are paid; only titles and scope are given) [Sources 38, 39]. Designing the tests is hard to leave to AI and is a job for people with domain knowledge (this article's commentary).
In METR's experiment the developers' impression of being "20% faster" and the measured "19% slower" were opposite [Source 5]. If you want AI to make analysis or scripting more efficient, measure working time and the number of errors for the same kind of work with and without AI and compare (commentary). Code written by AI for analysis also bears on the reproducibility of research data. Save together the code version, the model and settings used, the input data, the output results, and the extent a person checked. Because models change in a short time in this field, the same prompt may not produce the same code in future (commentary).
5. Agents that operate browsers and screens
(1) How it works: look at the screen, operate by coordinates
The Claude "computer use" tool documentation describes giving the model screenshot, mouse and keyboard control of a desktop environment. The model returns tool calls (such as coordinates to click), the application executes them in its own environment, and returns results and a screenshot, in a round-trip loop [Source 28]. Gemini API's Computer Use is similar, supporting browser, mobile and desktop environments; the documentation describes an intent attached to each action to explain the model's reasoning, and an internal safety system that classifies each action as regular, requiring confirmation, or blocked (the page states last updated 13 October 2026) [Source 29]. OpenAI's Computer-Using Agent (dated 23 January 2025) combines GPT-4o's vision with reinforcement-learned reasoning to operate GUIs, the buttons, menus and text fields, as a person does, and says it asks the user for confirmation on sensitive actions such as entering login details or responding to CAPTCHA forms [Source 27]Sourced.
(2) The difficulty of pinning down "where on the screen" (grounding)
"Grounding", identifying where on the screen to act, is a central challenge. Set-of-Mark (Yang et al.) is a prompting method that partitions an image into regions and overlays marks such as alphanumerics, masks or boxes, which brings out the visual grounding ability of GPT-4V [Source 26]. UI-TARS (Qin et al.) is a "native GUI agent" model that takes only screenshots as input and performs keyboard and mouse operations, reporting 24.6 with 50 steps and 22.7 with 15 steps on OSWorld (at the time of the paper) [Source 25]Sourced.
(3) Benchmarks, and how they keep moving
| Benchmark (source) | Content (as described) | Reported value (at publication) | Cautions |
|---|---|---|---|
| Mind2Web (Deng et al., 2023) [Source 22] | More than 2,000 tasks from 137 websites across 31 domains, with crowdsourced action sequences | The authors say there is still substantial room to improve | Raw HTML is too large, so the paper proposes filtering with a small LM first |
| WebArena (Zhou et al., 2023) [Source 24] | Long tasks on working sites from four domains | Best GPT-4-based agent 14.41%; humans 78.24% | OpenAI's CUA reported 58.1% (January 2025, under different settings) [Source 27] |
| VisualWebArena (Koh et al., 2024) [Source 23] | Web tasks that need visual information | Analyzes the limits of text-only agents and the gaps of multimodal agents | — |
| OSWorld (Xie et al., 2024) [Source 20] | 369 tasks in real computer environments (Ubuntu, Windows, macOS), with execution-based evaluation scripts | Humans over 72.36%; the best model 12.24%. The main weaknesses are GUI grounding and operational knowledge | OSWorld-Verified was released in July 2025 and OSWorld 2.0 on 26 June 2026 (per the official page) |
| OSWorld 2.0 (Yuan et al., 2026) [Source 21] | 108 long workflows; a human median of about 1.6 hours; an average of 318 tool calls (about 30 in OSWorld 1.0) | For the best model, 20.6% completion at 500 steps (partial score 54.8%) | The paper sums up failures as less about basic operation than losing track of constraints, missing information that arrives mid-task, guessing rather than asking, and skipping verification |
All Sourced (descriptions in each paper or page). Numbers are values at publication. OpenAI's CUA values are OSWorld 38.1%, WebArena 58.1% and WebVoyager 87% (humans 72.4% on OSWorld and 78.2% on WebArena), as stated on the company's January 2025 page.
Our calculation The OSWorld 2.0 paper says a task takes an average of 318 tool calls, against about 30 in OSWorld 1.0. If each operation succeeds independently with probability r, the chance that all succeed is r to the power n.
- r = 0.99: 0.9930 = about 74% for 30 calls, and 0.99318 = about 4.1% for 318 calls
- r = 0.999: about 97% for 30 calls and about 73% for 318 calls
Assumptions and limits: this idealizes operations as independent and failure as unaffordable. In practice some operations can be corrected partway and some failures do not affect the result, and the paper's completion rate (20.6%) is a separate measurement from this calculation. The real per-operation success rate r was not checked.
(4) Limits the documentation lists
The Claude documentation lists as limits of computer use: (1) it can be slower than direct human operation, so focus on use cases where speed is not critical (background information gathering, automated software testing) in trusted environments; (2) the model can make mistakes or hallucinate when outputting specific coordinates; (3) it may make mistakes in selecting tools or take unexpected actions to solve problems; and (4) reliability can be lower with niche applicationsSourced [Source 28].
6. Permissions and sandboxes
For an agent that has screen operation and code execution, damage grows when instructions coming from outside combine with excessive permissions. The countermeasures shown by primary sources are sorted below by layer (the layers are this article's own).
| Layer | Idea of the countermeasure (as the source describes it) | Source |
|---|---|---|
| Isolating the execution environment | Run in a dedicated virtual machine or container with minimal privileges to prevent direct attacks on the system or accidents. The Gemini documentation likewise asks for execution in a sandboxed VM or container | [Source 28] [Source 29] |
| Filesystem and network boundaries | On top of OS features, put both filesystem isolation (only the chosen directories) and network isolation (only approved servers). Anthropic says either alone lets a compromised agent exfiltrate files such as SSH keys or escape the sandbox | [Source 30] |
| Limiting data and destinations | Do not give access to sensitive data such as account login information. Limit internet access to an allowlist of domains | [Source 28] |
| Human confirmation | A person confirms decisions with meaningful real-world consequences and tasks requiring consent (accepting cookies, financial transactions, agreeing to terms). Gemini classifies actions as regular, require-confirmation or blocked, with policy categories such as financial transactions, sensitive-data modification, communication tools, account creation, data modification, consent management and legal terms | [Source 28] [Source 29] |
| Permission modes | Manual mode starts read-only and asks before edits and execution. The working-directory boundary is a confirmation mechanism, and an approved command can write wherever the user can. Asking every time causes approval fatigue and people stop reading | [Source 31] [Source 30] |
| Preparing for instructions arriving through the screen or external data | The Claude documentation says instructions on web pages or in images may be followed even when they conflict with the user's. A classifier layer scans screenshots, but it may not suit use cases with no human in the loop, and the precautions above stay important even with the classifier. Gemini offers screenshot scanning as an opt-in feature | [Source 28] [Source 29] [Source 33] |
| Preventing excessive agency | Avoid excessive functionality, excessive permissions and excessive autonomy (OWASP LLM06:2025) | [Source 32] |
All Sourced (descriptions in each source). The Claude Code and Claude documentation and Anthropic's blog are the developer's own descriptions, not a comparison with other companies' products.
General software-development frameworks also help with the safety of AI-written code. NIST SP 800-218 (SSDF) presents a core set of secure software development practices for reducing the risk of software vulnerabilities, and SP 800-218A (July 2024) is a community profile that adds practices specific to the development of generative AI and dual-use foundation modelsSourced [Sources 34, 35]. These are mainly practices for those who build models and software; this article did not confirm that they directly define "a procedure for accepting code written by an AI agent".
Avoid connecting AI-written code directly to measuring instruments or control systems; test in a verification environment first, and move only a version a person has reviewed into production (as in Fig. 6, keeping the production environment for instrument control physically apart from the verification one). It is also tempting to use a screen-operation agent for legacy instrument software that has no API, but coordinate errors and unexpected operations could affect instrument conditions or samples. If you do, use an isolated virtual machine, give no login credentials, start with uses where speed is not critical, and have a person confirm irreversible operations (overwriting conditions, deleting data) [Sources 28, 32]. Where possible, prefer the official control interface provided by the instrument maker (this article's commentary).
7. Limits, and what could not be confirmed
(1) Performance of current models
The benchmark numbers here are values for the models and settings at the time of each paper or report. We have not covered results for each company's latest models as of 15 October 2026, or current leaderboard positions. OpenAI stopped reporting SWE-bench Verified in February 2026, and this article does not rank by results of that time. The OSWorld 2.0 paper describes other companies' models' scores differently between versions (about 13% and about 14%), so those figures are not quoted.
(2) Developers' statements
Anthropic's internal reduction in permission prompts (84%), OpenAI's CUA scores, and each company's documentation of safety features are the developers' own statements; independent reproduction was not checked. Independent verification of the validity of SWE-Bench Pro is thought to be continuing, but this article stays within what primary sources could confirm.
(3) Retrieving sources
Three OpenAI pages (the SWE-bench Verified announcement, the 2026 report and Computer-Using Agent) refused automated retrieval, so their full texts were checked in Internet Archive copies. For ISO standard pages, titles, editions and dates were checked from the official-page entries shown in search results, and the texts were not checked. For ISO/IEC TS 42119-2:2025 only the title was checked.
(4) Standards
An international standard that directly defines acceptance of AI-written code or the evaluation of coding agents could not be found. ISO/IEC 25010:2023 and 29119-1:2022 are standards for a general quality model and testing concepts of software.
(5) What is not covered
The Model Context Protocol and agent interoperability, the details of evaluating and securing agents, and the general mechanisms of AI agents are covered in other explainers in this series. Feature, price or ranking comparisons of particular products (Claude Code, Copilot, Cursor, Codex and so on) are not covered. Concrete procedures for attacks or for evading security are not written. Copyright and licensing questions around code-generating AI are outside this article.
- The unit of work is widening. From completing a function to a repository's issues, and to long work at a terminal or screen. The wider it gets, the wider the scope a person must check and the impact of failureSourced
- Benchmarks move with quality. SWE-bench values fell with solution leakage and weak tests and rose when flawed tasks were removed, and OpenAI stopped reporting Verified in February 2026Sourced / Our calculation
- Impressions and measurements differ. METR's 2025 experiment found a 19% lengthening, and its 2026 update says the effect is "not measured reliably". Measure productivity on your own workSourced
- Screen operation is slow, mislocates coordinates and risks following instructions inside the screen. Even the best completes about a fifth of long tasks in OSWorld 2.0 (2026)Sourced
- Keep permissions minimal, isolate, and have a person confirm irreversible operations. Both filesystem and network boundaries are neededSourced
8. How to read evaluations and standards
Evaluating coding agents and screen operation has four sides: (1) task achievement (tests pass or fail, task completion), (2) productivity such as working time, (3) quality and security of the output code, and (4) safety of operation (permissions and isolation). Benchmark values move a great deal with the quality of problems and tests, leakage into training data, and version and settings; standards for software in general form a base, but a standard that directly addresses AI agents could not be found.
| What is evaluated | Representative benchmarks and methods (name, source) | Standards and guidance (ISO/IEC, NIST and others) | Cautions when reading |
|---|---|---|---|
| Function-level code generation | HumanEval (Chen et al.) [Source 1], MBPP (Austin et al.) [Source 2], LiveCodeBench (contamination-free, adds problems over time) [Source 14] | Not found | pass@k depends on the number of attempts k, and success rates differ between problems (the calculation in Section 2). The LiveCodeBench paper points out that existing benchmarks such as HumanEval and MBPP may be overfit Sourced |
| Resolving repository issues | SWE-bench (2,294 problems) [Source 8], SWE-bench Verified (500 samples) [Source 9], SWE-Bench Pro (1,865 problems) [Source 13], SWE-Bench+ (analysis of flaws) [Source 11], UTBoost (test augmentation) [Source 12] | Not found | The meaning of a value changes with version, scaffold and settings. OpenAI stopped reporting Verified in February 2026, citing contamination and flawed tests [Source 10]. Do not trust rankings as they stand Sourced |
| Long terminal work and real projects | Terminal-Bench 2.0 (89 tasks) [Source 16], SWE-Lancer (over 1,400 tasks, worth $1 million in payouts) [Source 15] | Not found | Values for the models at publication. The tasks are selected and not representative of real work as a whole Sourced |
| Productivity (working time) | The Copilot controlled experiment (55.8% faster) [Source 4], METR's controlled experiment (2025: 19% longer) [Source 5], METR's update of February 2026 (unreliable because of selection effects) [Source 6] | Not found | Results can reverse with the kind of task, participants' experience and the generation of tools. Impressions of time saved differ from measurements Sourced |
| Security of output code | Scenario-based evaluation by Pearce et al. (about 40% of 1,689 programs across 89 scenarios vulnerable; a 2021 tool) [Source 7] | NIST SP 800-218 (SSDF, February 2022) [Source 34] and SP 800-218A (development of generative AI and dual-use foundation models, July 2024) [Source 35]. Recommended practices; they do not directly define a procedure for accepting AI-generated code | Do not apply values for an old tool to the present. Have a person review the code and pass it through tests and static checks (commentary) Sourced |
| Operating screens and browsers | Mind2Web [Source 22], WebArena [Source 24], VisualWebArena [Source 23], OSWorld (369 tasks) [Source 20], OSWorld 2.0 (108 tasks) [Source 21] | Not found | The baseline of values changes with each updated version (OSWorld-Verified, OSWorld 2.0). Human performance changes with conditions Sourced |
| Safety of operation | Evaluation of agent hijacking (NIST CAISI technical blog) [Source 33] | OWASP LLM06:2025 Excessive Agency (an industry community document, not a law or a standard) [Source 32] | Evaluate adaptively and over multiple attempts. The design of permissions is central Sourced |
| General frameworks for software quality and testing | (standards, not benchmarks) | ISO/IEC 25010:2023 (product quality model; nine characteristics, with safety newly added) [Source 37]. ISO/IEC/IEEE 29119-1:2022 (general concepts of software testing) [Source 38]. ISO/IEC TS 42119-2:2025 (overview of testing AI; title only checked) [Source 39] | General frameworks that can be used for AI-written code too, but they do not directly define the evaluation of AI agents. The texts are paid and were not checked Sourced |
| AI management and risk | (a framework, not a benchmark) | ISO/IEC 42001:2023 (AI management system) [Source 40]. NIST AI RMF 1.0 (NIST AI 100-1, January 2023) [Source 36] | The NIST framework is intended for voluntary use. ISO/IEC 42001 is a standard for an organization's management system and does not guarantee the safety of an individual agent Sourced |
The organization of the table is this article's own (commentary). The texts of ISO/IEC standards are paid and were not checked. Checked 15 October 2026.
Three cautions for materials engineers
- Numbers change with settings, version and evaluator. Even under the same "SWE-bench", the meaning differs with the version, the scaffold and the evaluation settings, and a developer's statement is not an independent evaluation. Do not read the value of a model at publication as the performance of a current model
- Saturation and contamination: the problems and solutions of public benchmarks may be mixed into training data. OpenAI reported that all the frontier models it examined could reproduce the gold patches [Source 10]. Because improvement may reflect exposure to the benchmark rather than ability, check with your own tasks and with non-public tests. Saturation and contamination of benchmarks in general are covered in this series' explainer on benchmarks and how to read AI evaluations
- Tell "requirement", "recommendation" and "industry community document" apart in standards and guidance. ISO/IEC 25010 and 29119 are frameworks for software in general, NIST's SSDF is a set of recommended practices, and OWASP's is an industry community document. None directly addresses AI agents. "Conforms to the standard" does not mean an agent is safe. Legally binding regulation is covered in this series' explainer on the EU AI Act and others
9. Glossary
- Coding agent
- An agent that reads and edits code, runs tests and looks at the results, advancing development work in a loop.
- pass@k
- The probability that at least one of k generations is correct.
- HumanEval / MBPP
- Benchmarks measuring function-level code generation.
- SWE-bench
- A benchmark of whether real GitHub issues can be resolved. It has versions such as Lite, Verified and Pro.
- ACI (agent-computer interface)
- The set of tools for interacting with software, designed for agents.
- Scaffold
- The program that wraps a model and manages tool calls and the loop.
- Contamination
- Evaluation problems or solutions being mixed into training data.
- Sandbox
- An isolated execution environment with restricted access to files and the network.
- Approval fatigue
- People ceasing to read what they approve because there are too many confirmations.
- Computer use
- A capability that has the model look at screenshots and operate a mouse and keyboard.
- Grounding
- Linking an instruction to a position or element on the screen.
- Prompt injection
- Disturbing an agent's behavior with instructions inside data or web pages.
- Excessive agency
- Functionality, permissions or autonomy larger than necessary (OWASP).
- SSDF
- NIST's Secure Software Development Framework (SP 800-218).
10. References (primary sources)
- Chen M. et al. "Evaluating Large Language Models Trained on Code", arXiv:2107.03374 (2021) — arxiv.org/abs/2107.03374
- Austin J. et al. "Program Synthesis with Large Language Models", arXiv:2108.07732 (2021) — arxiv.org/abs/2108.07732
- Li Y. et al. "Competition-Level Code Generation with AlphaCode", arXiv:2203.07814 (2022) — arxiv.org/abs/2203.07814
- Peng S. et al. "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", arXiv:2302.06590 (2023) — arxiv.org/abs/2302.06590
- Becker J. et al. (METR) "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", arXiv:2507.09089 (2025) — arxiv.org/abs/2507.09089
- METR "We are Changing our Developer Productivity Experiment Design" (dated 24 February 2026). Checked 15 October 2026 — metr.org/blog/2026-02-24-uplift-update
- Pearce H. et al. "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions", arXiv:2108.09293 (2021) — arxiv.org/abs/2108.09293
- Jimenez C. E. et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", arXiv:2310.06770 (2023) — arxiv.org/abs/2310.06770
- OpenAI "Introducing SWE-bench Verified" (dated 13 August 2024). The official page blocks automated fetching, so the full text was checked in an Internet Archive copy — openai.com/index/introducing-swe-bench-verified
- OpenAI "Why SWE-bench Verified no longer measures frontier coding capabilities" (dated 23 February 2026; URL slug why-we-no-longer-evaluate-swe-bench-verified). The full text was checked in an Internet Archive copy — openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
- Aleithan R. et al. "SWE-Bench+: Enhanced Coding Benchmark for LLMs", arXiv:2410.06992 (2024) — arxiv.org/abs/2410.06992
- Yu B. et al. "UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench", ACL 2025, arXiv:2506.09289 — arxiv.org/abs/2506.09289
- Deng X. et al. "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?", arXiv:2509.16941 (2025) — arxiv.org/abs/2509.16941
- Jain N. et al. "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code", arXiv:2403.07974 (2024) — arxiv.org/abs/2403.07974
- Miserendino S. et al. "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?", arXiv:2502.12115 (2025) — arxiv.org/abs/2502.12115
- Merrill M. A. et al. "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces", arXiv:2601.11868 (2026) (the abstract describes "Terminal-Bench 2.0") — arxiv.org/abs/2601.11868
- Yang J. et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering", arXiv:2405.15793 (2024) — arxiv.org/abs/2405.15793
- Xia C. S. et al. "Agentless: Demystifying LLM-based Software Engineering Agents", arXiv:2407.01489 (2024) — arxiv.org/abs/2407.01489
- Wang X. et al. "OpenHands: An Open Platform for AI Software Developers as Generalist Agents", arXiv:2407.16741 (2024) — arxiv.org/abs/2407.16741
- Xie T. et al. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments", arXiv:2404.07972 (2024) — arxiv.org/abs/2404.07972
- Yuan M. et al. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks", arXiv:2606.29537 (2026) — arxiv.org/abs/2606.29537
- Deng X. et al. "Mind2Web: Towards a Generalist Agent for the Web", arXiv:2306.06070 (2023) — arxiv.org/abs/2306.06070
- Koh J. Y. et al. "VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks", arXiv:2401.13649 (2024) — arxiv.org/abs/2401.13649
- Zhou S. et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents", arXiv:2307.13854 (2023) — arxiv.org/abs/2307.13854
- Qin Y. et al. "UI-TARS: Pioneering Automated GUI Interaction with Native Agents", arXiv:2501.12326 (2025) — arxiv.org/abs/2501.12326
- Yang J. et al. "Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V", arXiv:2310.11441 (2023) — arxiv.org/abs/2310.11441
- OpenAI "Computer-Using Agent" (dated 23 January 2025). The official page blocks automated fetching, so the full text was checked in an Internet Archive copy — openai.com/index/computer-using-agent
- Anthropic "Computer use tool" (Claude Platform Docs). Checked 15 October 2026 — platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
- Google "Computer Use" (Gemini API; page states last updated 13 October 2026). Checked 15 October 2026 — ai.google.dev/gemini-api/docs/computer-use
- Anthropic "Beyond permission prompts: making Claude Code more secure and autonomous" (dated 20 October 2025). Checked 15 October 2026 — anthropic.com/engineering/claude-code-sandboxing
- Anthropic "Security" (Claude Code Docs). Checked 15 October 2026 — code.claude.com/docs/en/security
- OWASP Gen AI Security Project "LLM06:2025 Excessive Agency" (a document of an industry community project; not a law or a standard). Checked 15 October 2026 — genai.owasp.org/llmrisk/llm062025-excessive-agency
- NIST CAISI "Technical Blog: Strengthening AI Agent Hijacking Evaluations" (dated 17 January 2025) — nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations
- NIST NIST SP 800-218, "Secure Software Development Framework (SSDF) Version 1.1: Recommendations for Mitigating the Risk of Software Vulnerabilities" (February 2022) — csrc.nist.gov/pubs/sp/800/218/final
- NIST NIST SP 800-218A, "Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile" (July 2024) — csrc.nist.gov/pubs/sp/800/218/a/final
- NIST NIST AI 100-1, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)" (January 2023) — doi.org/10.6028/NIST.AI.100-1
- ISO/IEC ISO/IEC 25010:2023, "Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model" (Edition 2, November 2023; nine characteristics, with safety newly added) — iso.org/standard/78176.html
- ISO/IEC ISO/IEC/IEEE 29119-1:2022, "Software and systems engineering — Software testing — Part 1: General concepts" (Edition 2, February 2022) — iso.org/standard/81291.html
- ISO/IEC ISO/IEC TS 42119-2:2025, "Artificial intelligence — Testing of AI — Part 2: Overview of testing AI systems" (title confirmed only from the official-page entry in search results) — iso.org/standard/84127.html
- ISO/IEC ISO/IEC 42001:2023, "Information technology — Artificial intelligence — Management system" (Edition 1, December 2023) — iso.org/standard/81230.html
11. Claim-to-source audit
| Claim in the text | Basis | Class |
|---|---|---|
| Codex (HumanEval 28.8%, GPT-3 0%, GPT-J 11.4%, 70.2% with 100 samples, stated limits); MBPP (974 tasks, 59.6% few-shot, human feedback halves the error rate); AlphaCode (top 54.3% on Codeforces; sampling and filtering) | Reference 1 https://arxiv.org/abs/2107.03374 Reference 2 https://arxiv.org/abs/2108.07732 Reference 3 https://arxiv.org/abs/2203.07814 | Sourced |
| The Copilot controlled experiment (HTTP server in JavaScript, 55.8% faster); the METR controlled experiment (16 developers, 246 tasks; forecast 24% shorter, post-hoc estimate 20% shorter, measured 19% longer); the METR update of February 2026 (selection effects; 18% faster [38% faster to 9% slower] for a subset of the original developers; 4% faster [15% faster to 9% slower] for new ones) | Reference 4 https://arxiv.org/abs/2302.06590 Reference 5 https://arxiv.org/abs/2507.09089 Reference 6 https://metr.org/blog/2026-02-24-uplift-update/ | Sourced |
| Pearce et al. (about 40% of 1,689 programs across 89 scenarios vulnerable; 2021 tool) | Reference 7 https://arxiv.org/abs/2108.09293 | Sourced |
| SWE-bench (2,294 problems, 12 repositories, Claude 2 at 1.96%); SWE-agent (ACI; 12.5% and 87.7%); Agentless (three phases; 32.00% on Lite at $0.70; flagged problematic tasks); OpenHands (an MIT-licensed platform); Terminal-Bench 2.0 (89 tasks; below 65%) | Reference 8 https://arxiv.org/abs/2310.06770 Reference 17 https://arxiv.org/abs/2405.15793 Reference 18 https://arxiv.org/abs/2407.01489 Reference 19 https://arxiv.org/abs/2407.16741 Reference 16 https://arxiv.org/abs/2601.11868 | Sourced |
| The SWE-bench Verified announcement (13 August 2024; 93 developers, 1,699 samples screened; 38.3% and 61.1% flagged; 68.3% removed; 500 samples; GPT-4o at 33.2%, more than double the original 16%; underestimation); SWE-Bench+ (32.67% leakage, 31.08% weak tests, 12.47% to 3.97%, over 94% before the knowledge cutoff); UTBoost (36 and 345 instances, 40.9% and 24.4%, 18 and 11 ranking changes); SWE-Bench Pro (1,865 problems, 41 repositories); OpenAI's report of 23 February 2026 (59.4% flawed in a 27.6% audit; reproduction of gold patches; 74.9% to 80.9%; stopping reports and recommending Pro) | Reference 9 https://openai.com/index/introducing-swe-bench-verified/ Reference 11 https://arxiv.org/abs/2410.06992 Reference 12 https://arxiv.org/abs/2506.09289 Reference 13 https://arxiv.org/abs/2509.16941 Reference 10 https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ | Sourced |
| LiveCodeBench (contamination-free; 400 problems; possible overfitting); SWE-Lancer (over 1,400 tasks worth $1 million; most unsolved) | Reference 14 https://arxiv.org/abs/2403.07974 Reference 15 https://arxiv.org/abs/2502.12115 | Sourced |
| Claude computer use (screenshots with mouse and keyboard; the loop in which the app executes and returns results; limits: latency, coordinate errors, tool selection, niche apps; security: dedicated virtual machine, no sensitive data, allowlist, human confirmation, instructions inside the screen); Gemini Computer Use (intent; three safety decisions; policy categories; screenshot scanning is opt-in; sandbox; last updated 13 October 2026); OpenAI's CUA (23 January 2025; OSWorld 38.1%, WebArena 58.1%, WebVoyager 87%; humans 72.4% and 78.2%; confirmation for sensitive actions) | Reference 28 https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool Reference 29 https://ai.google.dev/gemini-api/docs/computer-use Reference 27 https://openai.com/index/computer-using-agent/ | Sourced |
| Set-of-Mark (overlaying marks); UI-TARS (a native GUI agent with screenshots only; OSWorld 24.6 and 22.7); Mind2Web (over 2,000 tasks, 137 sites, 31 domains); VisualWebArena; WebArena (14.41% and 78.24%); OSWorld (369 tasks; humans over 72.36%, best 12.24%; release of OSWorld-Verified and OSWorld 2.0); OSWorld 2.0 (108 workflows; median 1.6 hours; 318 tool calls on average; best completion 20.6%; the analysis of failures) | Reference 26 https://arxiv.org/abs/2310.11441 Reference 25 https://arxiv.org/abs/2501.12326 Reference 22 https://arxiv.org/abs/2306.06070 Reference 23 https://arxiv.org/abs/2401.13649 Reference 24 https://arxiv.org/abs/2307.13854 Reference 20 https://arxiv.org/abs/2404.07972 Reference 21 https://arxiv.org/abs/2606.29537 | Sourced |
| Anthropic's sandboxing (20 October 2025; approval fatigue; both filesystem and network isolation; 84% fewer permission prompts in internal use); Claude Code security (manual mode starts read-only; the working-directory boundary is a permission prompt); OWASP LLM06:2025; NIST CAISI hijacking evaluation; NIST SP 800-218 (February 2022) and SP 800-218A (July 2024) | Reference 30 https://www.anthropic.com/engineering/claude-code-sandboxing Reference 31 https://code.claude.com/docs/en/security Reference 32 https://genai.owasp.org/llmrisk/llm062025-excessive-agency/ Reference 33 https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations Reference 34 https://csrc.nist.gov/pubs/sp/800/218/final Reference 35 https://csrc.nist.gov/pubs/sp/800/218/a/final | Sourced |
| ISO/IEC 25010:2023 (Edition 2, November 2023; nine characteristics; safety added); ISO/IEC/IEEE 29119-1:2022 (Edition 2, February 2022); ISO/IEC TS 42119-2:2025 (title only); ISO/IEC 42001:2023; NIST AI 100-1 (January 2023). Numbers, titles and editions follow official pages; ISO full texts were not checked | Reference 37 https://www.iso.org/standard/78176.html Reference 38 https://www.iso.org/standard/81291.html Reference 39 https://www.iso.org/standard/84127.html Reference 40 https://www.iso.org/standard/81230.html Reference 36 https://doi.org/10.6028/NIST.AI.100-1 | Sourced |
| The mismatch between pass@k and independent trials (1 - 0.712^100 is about 1 versus 70.2% in the paper); the fall in SWE-Bench+ (about 68%, ratio about 0.32); the Verified ratio (about 2.1); the gap between METR's estimate and measurement (about 39 points); success over long tasks (0.99^30 = about 74%, 0.99^318 = about 4.1%, 0.999^30 = about 97%, 0.999^318 = about 73%) | Our calculation: simple arithmetic on values stated in the papers and reports, including an idealization of independent, identical success rates; examples under different settings are not for comparison. The actual per-action success rate r was not checked | Our calculation |
| Results for the latest models; independent reproduction of developers' statements; a standard directly defining the acceptance of AI-written code or the evaluation of coding agents; the full texts of ISO standards; independent verification of SWE-Bench Pro | Could not be confirmed in the primary sources accessible within this article's scope, so not stated | Commentary |
| The readings "tests are the specification", "measure rather than trust impressions, and keep versions, inputs and results of analysis code", "do not connect directly to production equipment, keep a verification environment apart, and isolate screen operation of legacy instrument software", the tables in Sections 4, 6 and 8, and the staged grouping by unit of work | This article's own organization and explanation based on published content; not views expressed by any company or institution | Commentary |
| Figs. 2 to 5 are explanatory drawings, not measured data; Figs. 1 and 6 are AI-generated image slots | A note by this article | Commentary |
Last updated 15 October 2026. Sources are limited to primary material (arXiv papers, developers' technical documents and blogs, official pages of METR, NIST and OWASP, and ISO/IEC standard pages). The texts of ISO/IEC standards are paid and could not be checked. Three OpenAI pages were checked in Internet Archive copies. Because the article includes readings and organization of its own, these are marked as Commentary and kept separate from sourced fact. Results for the latest models, independent reproduction of developers' statements, a standard directly defining acceptance of AI-written code or the evaluation of agents, and the texts of the standards could not be confirmed and are not stated here. Benchmark numbers are for the time of publication and the settings given, and do not show which model or product is better. Concrete attack procedures are not given. Figs. 2 to 5 are vector drawings; Figs. 1 and 6 are AI-generated image slots (none of them shows a real measurement result or a physical product).