Momentic Mo QA Bench Explained: AI QA Agent Finds 269 of 1,396 Bugs — But It Graded Its Own Homework

on

momentic mo qa bench explained — a hand holding a smartphone, illustrative image

Illustrative image: a hand holding a smartphone — the kind of app Mo tests by actually using it. This is not a Momentic screenshot. Image: Wikimedia Commons, CC BY 2.0.

TL;DR — Momentic Mo QA Bench explained in one line: Momentic's AI QA agent Mo found 269 of 1,396 labelled bugs in an hour of testing real web apps — 3.1 times more than OpenAI's Codex and 4.6 times more than Claude Code — but Momentic built the test, ran it and graded it, so the headline needs its warning label attached.

📌 KEY HIGHLIGHTS

  • The new test: QA Bench, published 2 November 2026, grades QA agents on 1,396 human-labelled bugs across four open-source web apps — Gitea, Metabase, Twenty and Saleor.
  • The headline: Mo matched 269.3 bugs per 60-minute pass (19.3% recall, 84.2% precision, $2.67 per bug). Codex with GPT-6.1 Sol managed 86.3; Claude Code with Opus 5.5 managed 59.0.
  • The catch: it is a first-party benchmark — Momentic designed, ran and graded it, and the bug list itself came partly from bug bashes by the agents being graded.
  • The honest read: even the winner missed over 80% of the labelled bugs. This is a progress report on AI testing, not proof that human QA is finished.
  • What Mo is: an AI QA engineer that explores a running app — web, iOS or Android — and reports only bugs it can reproduce, with a recording and steps.

Momentic Mo QA Bench explained: what happened?

On 2 November 2026, Momentic — a Y Combinator Winter 2024 startup — published QA Bench alongside a Launch HN post for Mo, its autonomous QA agent. Mo was unveiled in September 2026; QA Bench is the measurement Momentic built because, in its words, no benchmark existed for the job it wanted to measure: pointing an agent at a running web application and asking it to find bugs.

The distinction matters. Static analysis and unit tests already catch many defects before code ships. The bugs that slip through are increasingly the kind you only see by using the product — a button that does nothing, a filter returning the wrong rows, a settings page showing a stack trace. QA Bench tests exactly that: agents explore a live app in a fresh sandbox, with a fixed time budget of 30 or 60 minutes per product area, and no access to the app's source code or to the labelled bug list.

The corpus is deliberately ordinary software, not toy demos: four widely used open-source applications with 192,600 GitHub stars between them, split into 20 product areas. Gitea contributes 11 areas and 476 bugs, Metabase 4 areas and 464 bugs, Twenty 4 areas and 374 bugs, and Saleor 1 area and 82 bugs — 1,396 labelled bugs in total. Each bug went through reproduction, independent checking by Claude Code and Codex, and human review before it entered the ground-truth set.

Momentic Mo QA Bench explained: the results, agent by agent

Ten agent configurations ran the suite, each area three times per budget (Devin ran once). An LLM judge — GPT-6.1 Sol at high reasoning effort, plus a human reviewer — matched each agent's reports against the labelled bugs. Here is the 60-minute table, using Momentic's published per-pass averages:

Agent configurationBugs matched (60 min)RecallPrecisionCost per bug
Mo (Momentic)269.319.3%84.2%$2.67
Codex, GPT-6.1 Sol + Playwright MCP86.36.2%86.0%$0.80
Claude Code, Opus 5.5 + Playwright MCP59.04.2%—$4.09
Devin27.01.9%84%$3.89

momentic mo qa bench explained — a data centre server room, illustrative image

Illustrative image: a data-centre server room — Mo runs its bug bashes on hosted browsers and devices, not on a tester's laptop. Image: Wikimedia Commons / BalticServers, CC BY-SA 3.0.

Two patterns stand out beyond the headline. First, time helps Mo more than its rivals: going from 30 to 60 minutes, its matches rose 52% (176.7 to 269.3), while Codex with GPT-6.1 Sol rose only 16%. Second, the browser tooling mattered almost as much as the model: Playwright MCP found 10% to 58% more bugs than the agent-browser tool for the same models. How Mo works explains part of the gap: it builds a self-organising graph of sub-agents as it explores, with specialised planner, locator and vision agents running in parallel, and a separate agent reproduces each suspected bug from a clean state before it is reported. It also integrates with Vercel and Cloudflare deploy previews, so a bug bash can run on every pull request, and it is driven from the terminal with a qa command-line tool.

Why Mo's lead comes with a warning label

Momentic is unusually open about the benchmark's limits, and they are worth repeating because they change what the numbers mean. First, this is a first-party evaluation: Momentic builds Mo and designed, ran and graded the benchmark. Second, the ground-truth bug list was seeded by bug bashes from Mo, Claude Code and Codex — three of the agents being graded — which risks favouring the kinds of bugs those agents already find. Third, Mo carried an extra burden the others did not: it had to produce a reproduction video and a written report for every bug, while rivals only had to identify them. Fourth, the published runs had exceptions to the no-internet rule: Codex's web search was on, and some agent-browser runs had open web access from the shell. And fifth, Devin ran each area only once, against three runs for everyone else.

There is also the number nobody headlines: 19.3% recall means Mo missed 80.7% of the labelled bugs. Codex, the cheapest per bug at $0.80 and the most precise at 86.0%, found just 6.2%. The honest summary is not "AI QA is solved" but "the best dedicated QA agent finds about a fifth of known bugs in an hour, and general coding agents find far fewer." That is still a real signal for teams drowning in AI-written code — verification is now the bottleneck, and a specialised agent beats pointing a coding agent at the app — but it is a long way from a replacement for human testers, who still cover local payment methods, real hardware and the judgement calls no benchmark scores.

Why it matters for World Tech readers

No India-specific data is attached to this launch, and this post does not invent any. The relevance is read-across, and it is strong: Indian engineering teams are among the world's heaviest users of AI coding tools, and every team that lets an agent write code inherits the same question QA Bench tries to answer — who checks the agent's work? Our GDPx benchmark explainer covered the same week's other measurement story, where no model cleared 12% on professional agent tasks. Together the two benchmarks describe one week in AI: models that write and act, measured against work people get paid for — and scoring far below what the marketing suggests.

For developers, the practical takeaway is narrower and more useful: if you already point a coding agent at a running app to find bugs, a purpose-built QA agent found several times more in this test, but at a higher cost per bug than Codex. If you do not test the running app at all, the 19.3% figure is a reminder that most bugs in a real product are still found the old-fashioned way — by using it.

Should you care?

✅ Worth watching, ⏳ not worth reorganising around yet. QA Bench is the first public yardstick for AI agents that test real apps, and Mo's lead is large enough to take seriously. But it is a vendor's own test, the winner still misses four out of five known bugs, and independent reproduction is the step that would turn a launch claim into a fact.

Availability & pricing

Mo is available self-serve through Momentic's platform, driven from its app composer, the qa CLI or a scheduled run; it tests web apps in Chromium browsers and mobile apps on iOS simulators and Android emulators, not on physical devices. Momentic has not published separate Mo pricing; its broader platform offers a free tier and pay-as-you-go plans from $125 a month plus usage credits, and the benchmark's $2.67-per-bug figure is a test cost, not a price list. Treat any budget based on it as an estimate until Momentic publishes Mo rates.

FAQs

What is Momentic Mo?

Mo is Momentic's AI QA engineer: you point it at a web address or a mobile build, and its agents plan test cases, explore the app's user flows in parallel, and report bugs they can reproduce, each with a recording and reproduction steps. It was unveiled in September 2026 by the Y Combinator Winter 2024 startup Momentic.

What is QA Bench?

QA Bench is the benchmark Momentic published on 2 November 2026 to measure QA agents: 1,396 human-labelled bugs across 20 product areas in four open-source apps — Gitea, Metabase, Twenty and Saleor. Each agent gets a fixed 30- or 60-minute budget per area in a fresh sandbox, and is scored on recall, precision and cost per bug found.

How many bugs did Mo find compared to Codex and Claude Code?

In a 60-minute pass, Mo matched 269.3 bugs (19.3% recall) against 86.3 for Codex with GPT-6.1 Sol and Playwright MCP (6.2%) and 59.0 for Claude Code with Opus 5.5 and Playwright MCP (4.2%). Codex was slightly more precise (86.0% vs 84.2%) and cheaper per bug ($0.80 vs $2.67).

Does Mo replace human QA testers?

No — and Momentic does not claim it does. Its own documentation says Mo replaces the repeat passes, not the people: human testers still cover local payment methods, real hardware and exploratory judgement. With even the best agent missing over 80% of labelled bugs in this test, Mo is a force multiplier for QA teams, not a substitute for them.

TAKEAWAY: AI can now write the code and, with Mo, start checking it too — but the first real test of AI QA agents says the checking is still only a fifth done. Watch for an independent rerun of QA Bench before treating 269 as gospel.

Follow World Tech on TechByCountry for tomorrow's 6 PM update.

📸 Get tech in the last 24 hours on Instagram — follow @_techinlast24hrs for daily posts (9 AM–9 PM) and reels at 12:30 PM & 8:30 PM.

Sources

Comments

Post a Comment