Illustrative image: code on a developer's screen. Image: Martin Vorel / Wikimedia Commons, CC BY-SA 4.0
📌 Key Highlights
- Co-Design Bench (arXiv:2610.35138) tests coding agents on 273 tasks drawn from 37,463 real user requests across 26,314 coding sessions.
- Best model GPT-6 Astra met 69.9% of each task's criteria on average — but fully solved only 26.4% of tasks.
- GPT-6 Astra fully solved 41% of multimodal editing tasks and under 18% of repository-wide tasks.
- On older benchmarks like WebCompass-edit, the same models score above 99% — a gap of more than 35 points.
- The harness matters as much as the model: Qwen3.8-27B scored 75% in one environment and 56% in another on larger overhauls.
What is Co-Design Bench?
Co-Design Bench is a new benchmark for AI coding agents, published as a preprint on 26 October 2026 by researchers from Alibaba Group and several Asian universities, including Fudan University, Zhejiang University, the Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University. The paper is titled "Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models".
Instead of inventing test problems, the team started from real life. They collected 37,463 distinct user requests from 26,314 real coding sessions and narrowed them down to 273 tasks — cases where users had to rephrase their request several times or roll back the agent's changes, meaning the tasks had already given AI agents trouble in practice. Each task pairs a request with a runnable, pre-edit web application and 2,331 hidden, expert-written acceptance criteria in total.
The tasks split into two tracks. Multimodal Editing (100 tasks) asks the agent to make localised edits using images, audio, video or documents as input — for example, pulling an amount from a receipt and updating every dependent total in the app. Repository Editing (173 tasks) asks for coordinated changes across a whole codebase. Ten model endpoints were tested under one shared agentic harness, so the comparison is apples to apples.
Grading is deliberately strict. One AI judge tests the finished app in a browser, then a second model re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app.
The headline result: 26.4% fully solved
OpenAI's GPT-6 Astra was the best of the ten models tested. It met 69.9% of each task's requirements on average — a strong score that sounds like the agent nearly finishes the job. The stricter number tells a different story: only 26.4% of tasks were fully satisfied, with every acceptance criterion met. In nearly three tasks out of four, at least one detail was left undone, so a human still has to clean up before the code can ship.
The split between the two tracks is stark:
| Test | Best model (GPT-6 Astra) — tasks fully solved |
|---|---|
| Co-Design Bench, all 273 tasks | 26.4% |
| Multimodal Editing track (100 tasks) | 41% |
| Repository Editing track (173 tasks) | Under 18% |
| Older benchmarks like WebCompass-edit | Above 99% |
Behind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max took the next spots, while Grok 4.6 landed in last place. On larger overhauls, the much smaller Qwen3.8-27B climbed to second — though the authors do not claim a general advantage for the model from that result. Throwing more compute at the problem did not fix it either: GLM-5.2 burned through nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse.
Illustrative image: coding in progress. Image: Wikimedia Commons, CC BY-SA 4.0
Why the old benchmarks got it wrong
The authors' core argument is that the tests the industry has relied on became too easy. When a coding agent can run, check and fix its own code in a closed loop, it can score above 99% on benchmarks like WebCompass-edit — and differences between models become almost impossible to measure. Co-Design Bench exposes a gap of more than 35 percentage points against those older tests.
Performance fell across all ten models as soon as a task combined three or four types of change at once — layout, design, interaction and content. Those intertwined requirements are the norm in real projects, not the exception. The benchmark also shows that the harness — the program through which an agent reads, writes and executes code — shapes results as much as the model itself. Qwen3.8-27B scored 75% in one harness and just 56% in another on larger overhauls, and which of two models came out ahead depended on the environment. Leaderboards that list only the model therefore tell only part of the story, and the authors recommend reporting the model and the working environment together.
An earlier study by the research organisation METR points the same way: the actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on the widely used SWE-bench Verified benchmark.
Why it matters for Indian developers
AI coding tools are now part of everyday workflows in Indian startups and IT teams — coding agents you can talk to and agentic IDEs included. This result does not say those tools are useless. It says the last stretch of a task — the final detail, the dependent total that must update everywhere, the edge case in a repository-wide change — is where agents still fail most often.
The practical takeaway is about process, not pessimism. A 70% complete change request is genuinely useful scaffolding, but it is not a finished pull request. Teams that treat an agent's "done" as done will ship the missing 30% as bugs. Human review, browser-level testing of the finished app, and checking the agent's work against a written list of acceptance criteria — exactly what Co-Design Bench grades — remain the parts that cannot be skipped.
Should you care?
Yes, if you write or review code with an AI agent. The benchmark is a reality check, not a reason to stop using these tools: GPT-6 Astra still does roughly 70% of the work on real change requests, and that is a large productivity gain. But the headline number to remember is 26.4% — the share of tasks the best agent finishes completely. Until that number rises, treat every agent output as a strong first draft and keep a human in the loop before it reaches production. ✅ Verdict: use the agents, verify the output.
FAQs
What is Co-Design Bench?
Co-Design Bench is a benchmark for AI coding agents built from 273 real user change requests drawn from 26,314 coding sessions. It tests whether agents can turn multimodal requests — text, images, audio, video or documents — into working revisions of existing web apps, graded by strict browser-based checks. The paper is on arXiv as arXiv:2610.35138.
How good are AI coding agents at real tasks?
According to Co-Design Bench, the best model fully completes only 26.4% of real-world tasks, even though it meets about 70% of each task's requirements on average. In most cases the agent handles the bulk of the work but leaves at least one detail unfinished, so human cleanup is still needed on nearly three tasks in four.
Which AI coding agent performed best in Co-Design Bench?
OpenAI's GPT-6 Astra led all ten models tested, with 69.9% of criteria met per task and 26.4% of tasks fully solved. Claude Opus 5 and Qwen3.8-Max took the next spots, while Grok 4.6 finished last.
Why do AI coding agents score 99% on some benchmarks and 26% on others?
Older benchmarks let agents run, check and fix their own code in a closed loop, which inflates scores above 99%. Co-Design Bench uses real user requests, strict browser-based grading and a second model that challenges positive verdicts — and it tests combined changes (layout, design, interaction and content together), where every model struggles.
Takeaway
AI coding agents are genuinely useful, but Co-Design Bench puts a hard number on their limit: the best agent in the world fully finishes only about one real change request in four. The gap between "70% of the requirements" and "done" is where human developers still earn their keep — review the output, test it in the browser, and check it against your own acceptance criteria before it ships.
If you use a coding agent every day, try this once: take one of your own recent change requests, run it through the agent, and check every requirement yourself instead of trusting its final message. The benchmark's lesson is that the last 30% is where the real work hides — and how Claude fixes its own notes between sessions shows the labs are already working on the memory side of the same reliability problem.
Sources
- Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models — arXiv:2610.35138 (primary source, submitted 26 Oct 2026)
- The Decoder — AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test (8 Nov 2026)
- Oossa — AI coding agents still need human cleanup on most real-world tasks (8 Nov 2026)
- Co-Design Bench project page — codesignbench.github.io
Comments
Post a Comment