Can You Trust the "Best AI Coder" Rankings? OpenAI Audited SWE-Bench Pro, Found ~30% of Tasks Broken, and Retracted Its Recommendation

1. The Day the "Best AI Coder" Ranking Wobbled

"Which is better at writing code, GPT or Claude?" Somewhere along the way, we all learned to answer this question the same way: by lining up benchmark scores. 88.7% on SWE-bench Verified. 69.2% on SWE-Bench Pro. Numbers reported down to the decimal point look objective, unarguable — a clean ranking of raw ability.

Then, in July 2026, something happened that shook that assumption at its foundation. OpenAI itself audited SWE-Bench Pro — the coding benchmark the industry had come to regard as the hardest and most realistic — and concluded that roughly 30% of its public tasks were flawed. OpenAI then formally retracted its own earlier recommendation that the community adopt the benchmark as a leading coding eval^[1].

This is not a routine bug report about one metric. It cuts to the credibility of the very numbers the whole industry currently trusts when it talks about the "best AI coder." In this article I dig into four questions, following the published record: (1) what exactly was broken, (2) whether the score gaps between GPT, Claude and Gemini are really capability gaps, (3) the dynamic of models being "trained to the test," and (4) how we might measure practical ability instead.

2. What Actually Happened: OpenAI Audited SWE-Bench Pro

SWE-Bench Pro is a software-engineering benchmark built by the data company Scale AI. It hands a model a real issue from a real open-source repository and scores it on whether it can produce code that passes the tests. It arrived as the successor to SWE-bench Verified, which top models had saturated and which had drawn concerns about contamination — training data leaking the answers. Pro's pitch was "harder and more realistic."

What OpenAI audited was Pro's 731-task public split. The method had three stages. First, an automated data-quality pipeline that mechanically analyzed model attempts and how the tests graded them. Second, Codex-based investigator agents with repository access, validating suspicious tasks one by one. Third, an independent review by five experienced software engineers.

The results were sobering. The automated pipeline flagged 200 tasks (27.4%) as broken; the human reviewers flagged 249 (34.1%). Under either estimate, roughly one in three tasks was unreliable as an evaluation item. OpenAI's research team put it plainly: "We retract our earlier recommendation to adopt SWE-Bench Pro"^[1].

The crucial point is that this is not a story about weak models — it is a story about a warped ruler. If the ruler is warped, then every ranking measured with it — is GPT ahead, is Claude ahead, where does Gemini land — loses some of its meaning.

3. Inside the "Broken" Tasks: Four Flaw Patterns

So what does "broken" concretely mean? OpenAI grouped the flaws into four broad types.

Tellingly, these four split into two opposing pairs. "Overly strict" and "low-coverage" are opposite failures of the tests — too harsh versus too lenient. "Underspecified" and "misleading" are opposite failures of the prompt — missing versus wrong. In other words the grading drifts in both directions at once, so a strong model can lose points while a sloppy solution scores full marks. Under those conditions, reading a score gap as a capability gap is hard to justify.

According to OpenAI, flaws like these are characteristic of benchmarks "assembled programmatically from repository history." Mass-produce tasks with thin human review and you get items that hold together mechanically but look broken to any working developer.

4. Why Pro Existed: A History of Benchmark Generations

To make sense of this, you have to appreciate how many times coding evals have turned over in a short span.

The original SWE-bench was a landmark, built from real GitHub issues and pull requests. But it also contained ambiguous and unsolvable tasks, so a hand-curated subset — SWE-bench Verified — was created. By early 2026, though, top models were clustering around 88% on Verified: the metric was saturated. Concerns also grew that eval tasks had leaked into training data. In February 2026, OpenAI effectively treated Verified as a metric that had run its course.

Into that vacuum stepped Scale AI's SWE-Bench Pro, offering larger repositories, longer fixes, and stricter tests — "closer to real work." Scores duly dropped, with even top models landing around 60–70%. Making the benchmark harder was, in itself, a healthy direction.

But this audit revealed that Pro was broken in a different way. In pushing up difficulty and scale to escape saturation and contamination, quality control failed to keep pace, and about a third of the tasks turned out defective. The irony is sharp: making a benchmark harder and making it measure correctly are two different problems.

5. Lining Up the Scores: The Verified–Pro Gap

To ground the discussion, here are representative scores reported across sources in the first half of 2026. Numbers move with measurement conditions and timing, so treat these as directional, not definitive.

The first thing to notice: on Verified, GPT and Claude sit within a rounding error of each other (0.1 points). Reading 88.7% vs 88.6% as "GPT wins" is very nearly meaningless. Verified is pinned to the ceiling and has lost the resolution needed to separate the leaders.

On the harder Pro, scores fall to 60–70% and the gaps between models look wider — which is exactly why the industry decided "the real ranking is settled on Pro." And yet a third of Pro was broken. So the very arena where we thought we were reading "true capability differences" was itself warped, on roughly one task in three.

6. Are the GPT / Claude / Gemini Gaps Really Capability Gaps?

Back to the core question: is a ranking's point gap really a gap in ability?

Think it through, and believing a gap requires at least three things to hold. First, the tasks themselves are correct — prompts clear, tests valid. Second, the grading is fair — it does not fail correct solutions or pass sloppy ones. Third, the measurement conditions match — same scaffold, same number of attempts, same constraints. This audit showed that the first two of those collapsed on 30% of the tasks.

Consider it concretely. If two models differ by 0.1 points, and that difference comes from how a handful of "happen-to-be-broken" tasks landed, then the ranking reflects luck, not skill. Worse, on a saturated metric like Verified, the few points that remain are concentrated in a small set of genuinely hard tasks — and if that small set is skewed or defective, the order can flip arbitrarily.

So, honestly: small gaps (within a few points) between top models are weak evidence of a capability difference. That is not the same as saying "GPT and Claude are equivalent." It means the number alone cannot tell you whether a meaningful gap exists. You earn the right to talk about a real difference only when a double-digit gap shows up across multiple independent benchmarks, reproducibly.

7. Trained to the Test: Goodhart's Law

There is a second, structural reason not to swallow scores whole: the dynamic by which models get built to fit the benchmark.

Economics has a famous line, Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." In a world where benchmark scores feed directly into product value and headlines, development naturally gravitates toward scoring high on that benchmark. This need not be cheating. Tuning the data mix, iterating on eval-like tasks, grinding down known failure modes — all are legitimate optimization, but stacked together they produce a model overfit to the metric. Call it "training to the test."

Nastier still is contamination: the eval tasks themselves leaking into the training data. Public benchmarks live on the internet, so a model trained at scale may effectively have "seen" parts of the answer key. When that happens, the score measures memory, not ability. That very concern is part of why Verified gave way to Pro in the first place.

Which is what makes OpenAI's recommendation pointed. The next generation of benchmarks, they argue, should be "built by experienced software developers with human oversight baked in from the start, rather than assembled programmatically from repository history"^[2]. Read the other way: the more a metric is mass-produced and automated, the more vulnerable it is to teaching-to-the-test and contamination.

8. How to Measure Practical Ability

If benchmarks can't be trusted at face value, how do we measure an AI's genuinely useful ability? There is no silver bullet, but the direction is clear: decompose a single score into a bundle of viewpoints.

First, measure on tasks close to your own work. Rather than a generic benchmark, assemble 10–20 representative problems from your own codebase, language and framework, let the model solve them, and have humans review. The information yield is orders of magnitude higher. Second, judge the process, not just the one-shot pass rate. Requirement comprehension, design soundness, added tests, self-correction after failure, response to review comments — evaluate the path, not only the artifact. Third, watch cost and speed together. Two models with the same pass rate can differ enormously in tokens consumed, wall-clock time, and retries — and that difference is real-world value.

Fourth, look at reliability — how hard it is to break. Run the same task several times: does the output vary wildly? Change the conditions slightly: does it fall apart? In practice, "always 80 out of 100" beats "occasionally brilliant, occasionally a disaster." A benchmark's average score hides exactly this variance.

In short, practical ability is not "can it solve one hard problem once." It is "can it keep solving the problems of my environment, at reasonable cost, stably, and in a verifiable form." No single leaderboard can ever express that.

9. And Yet Benchmarks Are Not Worthless

By now you might want to conclude "benchmarks are all a sham" — but that overshoots. If anything, this episode confirmed something healthy: the evaluation community can self-correct. Doubt the metric, audit it, publish the flaws, retract the recommendation — that sequence is a scientific posture.

Benchmarks retain value as a coarse filter. Large gaps — "80% versus 40%" — are meaningful signals even with some noise. For roughly comparing, on common ground, whether a new model clearly jumped over its predecessor, they remain useful. The problem is not the existence of benchmarks; it is the habit of deifying sub-decimal gaps as a ranking of raw ability.

OpenAI itself did not say "don't use SWE-Bench Pro." Its advice was to "stop treating the scores as a clean measure of coding capability, and if you keep using it, scrutinize the results carefully before publishing them"^[3]. That is not a rejection of metrics; it is a call for maturity in how we use them.

10. A Practical Checklist for Reading the Numbers

Finally, here is a usable yardstick for the next time you read an AI ranking or a benchmark article.

• Look at the size of the gap. Treat differences within a few points as "roughly equal." Only double-digit gaps count as a trend.
• Don't trust a single metric. Is the model consistently on top across several independent benchmarks? Discount a first place that rests on one metric alone.
• Suspect saturation. If the leaders all cluster at high scores (say, the high 88s), that metric has already lost its resolution.
• Check the conditions. Same scaffold (agent setup), same number of attempts, same constraints? Different conditions mean the scores aren't comparable.
• Suspect contamination. An oddly high score on a public benchmark can't be distinguished from memorization.
• Run a small test yourself. In the end, a handful of tasks close to your own work speaks louder than any ranking.

11. Conclusion: To Doubt Is Not to Discard

What the July 2026 SWE-Bench Pro audit forced into the open is an inconvenient truth: the numbers the AI industry believes in are not as clean as we think. A third of the metric considered most realistic was broken, and the very lab that had championed it retracted its recommendation. The fragility of reading small gaps between GPT, Claude and Gemini as a "ranking of ability" was made vividly visible.

But the real lesson is not pessimism. Doubt the metric, audit it, publish, fix it — that healthy self-correction is exactly how AI evaluation inches closer to science. What is asked of us is not to throw the numbers away, but to doubt them correctly and use them correctly. Look at the size of the gap. Bundle multiple metrics. Suspect saturation. And in the end, run a small test in your own environment.

As long as we keep asking "can you trust that number?", rankings remain not an oracle that rules us, but one tool among many that helps us decide. No matter how smart the AI becomes, deciding what to measure, and how, is still a human job.

---

参考文献 / References

1. OpenAI Retracts SWE-Bench Pro After Finding 30% of Tasks Broken — AlphaSignal
2. OpenAI Retracts Recommendation To Use SWE Bench Pro As Coding Eval Over 30% Broken Tasks — OfficeChai
3. OpenAI wants new benchmarks to replace "broken" SWE-Bench Pro — The Stack
4. OpenAI Flags Major Flaws in SWE-Bench Pro — StartupHub.ai

References

1. [1]OpenAI's audit covered SWE-Bench Pro's 731-task public split via three stages: (1) an automated data-quality pipeline, (2) Codex-based investigator agents with repository access, and (3) independent review by five experienced software engineers. The automated pipeline flagged 200 tasks (27.4%) and the human reviewers flagged 249 (34.1%) as broken. ↩
2. [2]OpenAI recommends that next-generation benchmarks be "built by experienced software developers with human oversight baked in from the start, rather than assembled programmatically from repository history." Mass-produced, automated metrics tend to outrun their own quality control, letting defective tasks slip in. ↩
3. [3]The scores in this article are representative values reported across sources in the first half of 2026 and vary with measurement conditions, agent scaffolding and timing. They are offered not to settle which model is superior, but as reference points for discussing the reliability of the metric itself. ↩