What Benchmarks Really Measure

Qwen3.8-27B is a comparatively compact model with 27 billion parameters. Its model weights are available under the Apache 2.0 license, which means it can also be run on your own infrastructure. On SWE-bench Pro it reaches 61.7%. Claude Opus 4.6, a large commercial model from Anthropic, appears in the same comparison table at 53.4%. [1][2]

At first glance, the conclusion seems surprisingly simple. If we compare coding agents and one model reaches 61.7 percent while another reaches 53.4 percent, the obvious interpretation is that Qwen3.8-27B must be the better coding agent. Perhaps technology evaluation no longer needs long discussions at all; perhaps we only need a table and the largest number in it.

Before drawing that conclusion, however, one rather unexciting question is worth asking: What does that percentage actually mean?

Interestingly, we do not even need to leave Qwen’s model card to find the first complication. The table really does show 61.7 for Qwen3.8-27B and 53.4 for a comparison entry that Qwen labels “Opus4.6 Max.” Anthropic itself reports the underlying result as 53.4% for Claude Opus 4.6; there is no basis here for treating “Opus 4.6 Max” as a separate Anthropic model variant. [1][2]

Directly below the table, Qwen also documents the evaluation setup. With the exception of the imported Opus score, the compared models were evaluated with a Claude Code harness, temperature=1.0, top_p=0.95, and a 256K context window. Qwen also states that problematic tasks were corrected and the baseline models rerun on this “refined benchmark.” The Opus result is explicitly the exception: Qwen reused Anthropic’s officially reported score rather than rerunning it in the same setup. [1]

The dataset source is also traceable. Qwen’s published evaluation metadata points to ScaleAI/SWE-bench_Pro, whose public test split contains 731 tasks. SWE-bench Pro as a whole is larger and also includes held-out and commercial sections. [1][6]

None of this makes the numbers wrong or suspicious. It simply means that two percentages are not automatically fully comparable merely because they appear next to each other in the same table. Even in our apparently trivial opening comparison, the model, harness, sampling settings, dataset variant, and origin of the comparison score all matter.

The problem is not the score. The problem is how we interpret the score.

What Is a Benchmark?

A benchmark is nothing mysterious. It is a standardized test, or a collection of tasks, used to measure and compare systems under conditions that are as well defined as possible.

It is useful to think of a benchmark more like an exam than a complete professional assessment. A mathematics exam can meaningfully measure how well someone solves certain mathematical problems. A very good result is relevant information. It still does not automatically tell us how well that person will work as an engineer, handle ambiguous requirements, document decisions, or evolve an existing system over ten years.

The same is true for coding benchmarks. A good benchmark can measure a particular software-engineering capability remarkably well under specific conditions. It does not follow that the benchmark measures every capability required in professional software development.

A benchmark is not a verdict on a model. It is a measuring instrument. Before comparing the number, we need to understand what that instrument actually measures.

SWE – Software Engineering

The abbreviation SWE now appears in many places: SWE-bench, SWE-bench Verified, SWE-bench Pro, SWE-Lancer, and SWE-rebench.

The expansion is much less exotic than the names might suggest:

SWE simply stands for Software Engineering.

The label signals that these benchmarks aim to examine more than generating a single function from a short problem statement. How closely they resemble real software engineering depends on the benchmark.

SWE-bench – From a Coding Problem to a Repository

Many classic coding benchmarks use largely isolated tasks. A model may receive a problem description and a function signature, generate an implementation, and then be evaluated with tests.

In simplified form:

problem description
↓
implement function
↓
run tests

That is useful, but it measures only part of what developers do in practice. In an existing project, the problem is rarely that someone hands us an empty function with a perfect description. We first need to discover which files matter, how the current system works, what change is actually intended, and which existing behavior must not break.

SWE-bench moved evaluation closer to existing software repositories. The original benchmark contains 2,294 tasks from twelve popular Python repositories. The tasks were derived from real GitHub issues and the corresponding changes. A system receives the repository state before the human fix together with the original problem description and must produce a patch that resolves the issue. The evaluation then checks whether previously failing tests pass and whether existing behavior remains intact. [3]

The basic idea therefore looks more like this:

repository state
+
GitHub issue
↓
inspect codebase
↓
find relevant location
↓
produce patch
↓
run tests
↓
resolved / unresolved

Compared with an isolated function, this is a substantial step toward software engineering. The system has to orient itself in unfamiliar code and integrate a change into an existing context.

It is still a controlled task. Nobody automatically evaluates whether the architecture will still be understandable in three years, whether a name matches the team’s domain language, or whether the patch fits into a long-term migration strategy. SWE-bench measures something much more realistic than pure code completion, but it still measures a defined slice of the profession.

What Does “61.7% Resolved” Mean?

On SWE-bench-style leaderboards, the proportion of tasks solved according to the benchmark’s evaluation rules is typically reported as % Resolved. [3]

A value of 61.7% therefore means, roughly:

Under the stated evaluation conditions, 61.7 percent of the tested benchmark tasks were classified as solved according to that benchmark’s rules.

That sounds almost trivial, but it is the most important translation in this entire article. The denominator of that percentage is benchmark tasks. The denominator is not every conceivable software task, every activity performed by a developer, or every situation that can occur inside one organization.

61.7% SWE-bench Pro therefore does not mean that the model can solve 61.7 percent of all software tasks. It does not mean the model is “61.7 percent of a software engineer,” nor that it replaces 61.7 percent of a developer position.

61.7% on SWE-bench does not mean 61.7% of a software engineer.

Why SWE-bench Verified Was Created

Using real GitHub projects has an obvious advantage: the tasks were not artificially constructed for a textbook benchmark. At the same time, it introduces a new problem. Real repositories, issues, dependencies, and tests were never written for the purpose of creating a scientifically clean measuring instrument.

The original SWE-bench tasks were collected largely automatically. Some later turned out to be ambiguous, poorly specified, or problematic because of their tests and environments. A functionally correct solution could fail an overly narrow test; in other cases the task description omitted information that a hidden test assumed. Differences in Python versions, operating systems, or dependency states could also affect reproducibility. [4][5]

OpenAI and the SWE-bench team therefore had 1,699 candidate tasks reviewed by experienced software engineers. Each task was independently evaluated by three experts. The result was SWE-bench Verified, containing 500 selected tasks. [4][5]

“Verified” in this context does not mean that somebody mathematically proved every possible evaluation to be perfect. It means humans reviewed the tasks and their evaluation and filtered out unsuitable cases.

That was a sensible improvement. The benchmark’s later history also shows how difficult it is to build a measuring instrument for software engineering that remains useful over time.

When a Successful Benchmark Becomes Part of the System

SWE-bench Verified became very successful. Model vendors published their scores, agent developers analyzed successful and failed trajectories, harnesses became better adapted to repository work, and the benchmark turned into an important reference point for coding capability.

That success gradually changes the conditions under which the benchmark operates.

Contamination

In benchmark evaluation, contamination refers to the risk that evaluation data, or highly similar information, was already available during a model’s training or post-training.

With SWE-bench, the issue is easy to understand. The original repositories are public, as are the issues and often the pull requests or commits containing the human solutions. If models are trained on large amounts of public code and technical communication, overlap with benchmark material cannot simply be ruled out.

That does not automatically mean the model “looks up” a memorized solution during evaluation. There is a major difference between incidental overlap in training data, learned project-specific knowledge, and reproducing a known solution. Contamination is first and foremost an evaluation risk, not an accusation of manipulation.

In February 2026, OpenAI published a new analysis of SWE-bench Verified and concluded that the benchmark had become too strongly affected by such effects and by remaining task problems to measure current frontier models cleanly. In the investigated tasks, all tested frontier models could in some cases reproduce original gold patches or highly specific problem information. OpenAI subsequently stopped reporting SWE-bench Verified scores for its own models. [4]

Other work now tries to address the same problem of static, publicly available software-engineering benchmarks. One approach is to continuously collect new tasks and separate evaluation temporally from older training data. [12][13]

Benchmark Overfitting

Contamination is not the same as benchmark overfitting.

A benchmark can become important enough that models, post-training, prompts, or agent harnesses increasingly optimize for exactly the capabilities the benchmark rewards. Nobody needs access to the concrete answers for that to happen. It can be enough to become better and better at the typical task structures, tool sequences, or strategies that perform well on the benchmark.

The school analogy works surprisingly well. If an exam has nearly the same structure for years and all previous exams are publicly available, students do not only learn mathematics. They also learn how to take that specific exam efficiently. Genuine knowledge may still be required, but exam performance becomes harder to separate from exam-specific preparation.

This is not an AI-specific problem. Machine learning has dealt with overfitting to datasets, benchmarks, and leaderboards for a long time. The more important a metric becomes, the stronger the incentive to optimize for exactly that metric.

A system can therefore improve substantially on one benchmark without the same improvement transferring proportionally to every other software-engineering task.

Why SWE-bench Pro Was Created

Scale AI developed SWE-bench Pro to push several of these limits further. The benchmark contains 1,865 tasks from 41 repositories. Of those, 731 belong to the public split, 858 to a held-out section, and 276 come from commercial codebases. The repositories cover Python, Go, JavaScript, and TypeScript. [6]

The tasks are also larger on average than many classic SWE-bench cases. Scale reports an average of 107.4 changed lines of code across 4.1 files. Because original commit messages and issues do not always provide enough context, human experts supplement the problem statements and requirements without prescribing the exact implementation. [6]

The different splits address different problems. The public split remains reproducible and openly evaluable. Held-out tasks are not published in full as a public benchmark dataset, making direct optimization on the exact evaluation tasks more difficult. The commercial tasks additionally come from private codebases, reducing the risk that those exact repository states and solutions were already present in public training data.

One distinction is important: held-out does not automatically mean “guaranteed never seen during training.” It initially means that evaluation data is withheld. How completely that excludes training overlap depends on the origin of the data and the training pipeline of the model being evaluated. Private commercial codebases provide a stronger separation here than benchmark tasks that are merely unpublished.

Held-out and private data therefore reduce contamination risks. They do not make a benchmark automatically error-free.

And Then SWE-bench Pro Was Audited Too

In July 2026, OpenAI published an audit of SWE-bench Pro. That is especially interesting because OpenAI had recommended moving from Verified to Pro only a few months earlier. [4][7]

For the new analysis, OpenAI examined the 731 tasks in the public SWE-bench Pro split. An automated analysis pipeline flagged 200 tasks, or 27.4 percent, as problematic. In the subsequent human annotation, 249 tasks, or 34.1 percent, were identified as having relevant problems. OpenAI therefore summarized its estimate cautiously as roughly 30 percent problematic or “broken” tasks and withdrew its earlier blanket recommendation of SWE-bench Pro. [7]

The reported issues included overly strict tests, underspecified prompts, insufficient test coverage, and, in some cases, misleading task descriptions. These figures come from OpenAI’s audit. They do not mean that exactly 30 percent of the entire SWE-bench Pro benchmark is objectively and indisputably unusable; even the difference between the automated and human assessments shows that judging the evaluation is itself work.

That is precisely why the episode is instructive. SWE-bench Pro did not become worthless because someone found problems in it. The audit instead demonstrates how difficult it is to make real software changes simultaneously realistic, reproducible, and automatically gradable.

Measuring real software engineering is itself a difficult software-engineering problem.

A very recent example can also be found in SWE-bench Multimodal. The original version contained 517 issues with visual information. On September 1, 2026, the SWE-bench team released version 2 with 480 tasks selected for reproducible evaluation. Known flaky or unreliable tests were removed, Docker environments were rebuilt to counter dependency and browser drift, and JavaScript grading and visual test-asset handling were made more robust. [14]

A benchmark is therefore less like a ruler carved in stone and more like a technical product that itself requires maintenance.

Model Score or System Score?

This brings us to a point that is especially easy to miss with coding agents. An agentic coding benchmark often does not measure the bare base model.

Between the model and the final patch sits an entire runtime environment. It determines which files can be searched, which tools are available, how shell commands are executed, how test results are returned to the model, how context is managed, and how many further attempts the agent may make.

As a mental model, we can think about the score like this:

This is obviously not a mathematical equation. The parts cannot simply be added together. It merely reminds us that the observed score emerges from the interaction of a whole system.

Our Qwen example demonstrates this unusually clearly. Qwen specifies not only the model but Claude Code as the harness, a 256K context window, concrete sampling parameters, and a corrected task state. [1]

If all we copy is 61.7%, we throw away a large part of the experiment description.

Harness – Giving the Model a Working Environment

We introduced the term Agent Harness in the previous article. It refers to the runtime and control layer around the model.

A harness can search files, structure tool calls, return shell output, manage context, organize iterations, and define when or how an agent finishes a task. Two agents using the same base model can therefore achieve different results.

This is not an annoying side effect that must always be removed from benchmarks. For a coding agent, the harness is part of the system. If better tooling lets a model find the relevant file faster or interpret a failed test more effectively, that is genuinely valuable in real use.

The problem begins only when we attribute the resulting system score exclusively to the base model.

Current leaderboards try to control this variable in some cases. For SWE-bench Verified, for example, the official site offers a Bash Only view in which models are compared inside the same mini-SWE-agent environment. [3] That does not remove every difference, but it standardizes part of the harness variation.

Pass@1 and Pass@k

Another small number attached to a benchmark can substantially change its meaning: k.

Pass@1 looks, in simplified terms, at success with a single attempt. Pass@k describes or estimates how often at least one of several generated candidates is successful. The concept became widely known through the HumanEval evaluation of Codex. [8]

If a system has some realistic chance of success on each attempt, producing several candidates naturally increases the chance of obtaining at least one successful result. Generating eight candidates and then selecting a successful one is therefore a different experiment from allowing exactly one attempt.

That does not make pass@8 worse than pass@1. In real products, generating several alternatives in parallel and then selecting one automatically or with human help may be entirely sensible. What matters is that we do not compare two numbers as though both systems had the same number of chances.

The k is therefore not a statistical footnote. It is part of the experimental setup.

Token and Compute Budget

The same applies to compute budget. An agent that works on a task with a few iterations and 50,000 tokens is operating under different conditions from one that is allowed millions of tokens, many tool calls, and many additional steps.

More compute is not inherently bad. If additional inference reliably produces better results and the real use case permits that budget, it may be a perfectly reasonable technical choice.

For benchmark comparison, however, we still need to know what resources each system was allowed to use. Otherwise we are comparing not only capabilities but also different amounts of permitted work.

The token-consumption analysis published on the SWE-rebench leaderboard (retrieved September 10, 2026) illustrates how large those differences can be. Token consumption across agent systems varies by more than a factor of 30 even though the systems are evaluated on the same task family. [12]

Whether that additional budget is economically worthwhile is another question. It gets its own place later in this series.

Even Infrastructure Can Add Points

In classic text benchmarks, the machine on which an answer is generated is often almost invisible. A coding agent, by contrast, actively works inside its environment. It installs dependencies, starts compilers, runs tests, and may create CPU- or memory-intensive processes.

Anthropic investigated this effect in early 2026 using Terminal-Bench 2.0. With the same model, the same harness, and the same task set, only the resource configuration was changed. The gap between the tightest and most generous tested infrastructure was six percentage points. Even across more moderate configurations, the spread remained close to two points. [9]

That does not mean every benchmark score is wrong by six points. It shows that runtime conditions are part of the experimental setup for agentic evaluations. A process killed because of a memory limit appears in the final score as an unresolved task just like a task where the agent made the wrong technical change.

When leaderboard differences are only one or two percentage points, statements such as “Model A clearly beats Model B” therefore deserve some restraint.

When the Benchmark Itself Changes

Terminal-Bench provides another useful example in 2026. Version 2.1 corrected 28 of the 89 tasks from Terminal-Bench 2.0 and introduced continuous validation. When the models were rerun, some scores changed substantially. Claude Opus 4.6 with Claude Code, for example, rose from 58.0 to 70.1 percent in the published comparison. [10]

The model did not suddenly become more intelligent. The benchmark changed in an attempt to measure more fairly and reproducibly.

The name “Terminal-Bench” alone is not enough: the official benchmark overview already lists Terminal-Bench 3.0, while the Qwen3.8 model card cited here still uses Terminal-Bench 2.1. [1][10] What matters for a comparison is the specific version and the specific experimental setup, not the assumption of a single “current” reference.

If someone quotes a “Terminal-Bench score,” they should therefore also quote the version and the setup. Scores from different versions and setups are not directly comparable.

Benchmark version numbers are not pedantry. They are part of the measurement.

A Higher Score Is Not Yet a Procurement Decision

Let us return briefly to the opening example. Qwen3.8-27B has 27 billion parameters; its model weights are available under Apache 2.0, and the model card describes several ways to run it on your own infrastructure. [1]

A decision-maker could therefore look at two numbers:

Qwen3.8-27B:      61.7%
Claude Opus 4.6:  53.4%

and conclude that a few internally operated GPU systems must clearly be the better choice than commercial cloud models.

The benchmark alone does not support that decision. Real procurement also depends on the actual task profile, required capabilities, data sovereignty, privacy, confidentiality, operations, concurrency, latency, context requirements, cost, and later review effort.

Local or self-hosted processing also has a more precise meaning than “the data stays in-house” sometimes suggests. Self-hosted inference can enable fully internal processing if tooling, telemetry, monitoring, retrieval, and all other involved components are also operated and configured accordingly.

Conversely, a higher general benchmark score does not imply that every development task should use the strongest available frontier model. A narrowly scoped and well-specified implementation task may be handled perfectly well by a smaller model, while other work may demand much stronger analysis or exploration.

Those are important decisions. They simply are not decisions that SWE-bench Pro measures.

Different Benchmarks Ask Different Questions

SWE-bench is therefore not bad because it does not represent all of software engineering. No single benchmark can reasonably do that.

The landscape becomes more useful when we treat benchmarks as different measuring instruments:

Benchmark	What does it roughly test?
SWE-bench Pro	changes to existing repositories based on realistic software-engineering tasks
Terminal-Bench	multi-step work in terminal environments using tools, programs, and system interaction
SWE-Lancer	real formerly paid freelance software tasks and technical decision tasks
LiveCodeBench	continuously refreshed algorithmic coding tasks with temporal separation to reduce contamination
SWE-rebench	continuously collected repository tasks for fresher software-engineering evaluation
SWE-bench Multilingual	SWE-bench-style repository tasks across multiple programming languages
SWE-bench Multimodal	repository issues in which visual information forms part of the requirement

Terminal-Bench expands the action space beyond repository patches. An agent works inside a terminal environment and may have to build software, change configuration, run programs, or interact with existing tools. The substantial revision from version 2.0 to 2.1 simultaneously demonstrates how difficult reproducible evaluation of such environments is. [9][10]

SWE-Lancer emphasizes a different dimension. The benchmark contains more than 1,400 real freelance tasks from Upwork associated with roughly one million US dollars in actual compensation. The tasks range from 50-dollar bug fixes to feature implementations originally worth 32,000 dollars. It also contains management tasks in which technical proposals must be evaluated. [11] The monetary framing makes it interesting for some economic questions, but it still is not a general measure of “the value of a developer.”

LiveCodeBench mainly addresses the time problem of static coding benchmarks. New tasks are continuously collected from programming competitions such as LeetCode, AtCoder, and Codeforces and tagged with their publication dates. Models can therefore be evaluated on problems released after a known training cutoff. [13] At the same time, the benchmark primarily measures algorithmic coding and related abilities such as code execution or self-repair. That is different from navigating a ten-year-old enterprise codebase.

SWE-rebench addresses freshness directly at the repository level. The original 2025 release collected more than 21,000 interactive Python tasks from over 3,400 repositories and used continuously refreshed tasks for more contamination-resistant evaluation. SWE-rebench V2 substantially expands the collection: the 2026 version describes more than 32,000 executable tasks from over 3,600 repositories and 20 programming languages, plus more than 120,000 additional tasks with installation information and metadata. [12] The core idea remains the same: a dynamic supply of fresh tasks makes it harder for one static dataset to become the industry’s permanent exam sheet.

SWE-bench Multilingual broadens language coverage. It contains 300 curated tasks from 42 repositories across nine programming languages: C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust. [15] That matters because a largely Python-based benchmark tells us relatively little about whether the same capabilities transfer reliably to other ecosystems.

SWE-bench Multimodal asks another question: what happens when the requirement is not fully contained in text? Screenshots of a bug, design mockups, wireframes, or diagrams are normal parts of frontend work. Version 2, released on September 1, 2026, contains 480 tasks selected for reproducible evaluation. [14]

None of these benchmarks is therefore “the best.” They ask different questions.

A Public Benchmark Is a Map, Not Your Terrain

Public benchmarks are extremely useful for obtaining an initial orientation. A model that trails clearly across several coding evaluations relevant to our use case sends a different signal from one that performs strongly across multiple measuring instruments.

For a particular organization, however, the decisive question may look very different.

Perhaps most of the work is not Python bug fixing but Angular migrations. Perhaps the repository is huge, the architecture highly modular, and one of the most important quality criteria is preserving existing boundaries. Perhaps agents are mainly expected to extend tests, analyze legacy code, or prepare changes in a proprietary .NET codebase.

In that case, internal evaluations can become far more relevant.

An internal eval does not have to become a new scientific world benchmark. Even a small, reproducible collection of representative tasks can show how different model-agent systems behave against your own reality: a typical bug fix, a feature, a refactoring, a migration, test additions, or repository analysis.

The same discipline still matters. What exact task does the system receive? Which repository state is used? Which tools are allowed? What budget does the agent receive? How is success judged? And who verifies that the internal evaluation actually measures what the organization cares about?

Public benchmarks help us place models. Internal evals help us prepare concrete decisions.

Back to 61.7 Percent

We started with two numbers:

Qwen3.8-27B      61.7%
Claude Opus 4.6  53.4%

After everything we have seen, those numbers are not less interesting. Qwen3.8-27B achieves a notable SWE-bench Pro result for a model of its size. The availability of its model weights and the option to run it on your own infrastructure make that result technically and strategically relevant. [1]

The numbers should simply feel less magical now.

We know that Qwen’s 61.7% is reported on ScaleAI/SWE-bench_Pro using Claude Code, specific sampling settings, a 256K context window, and a corrected task state. We also know that the 53.4% for Claude Opus 4.6 in the same Qwen table did not come from the same Qwen run but was imported as Anthropic’s officially reported value. [1][2]

We know that SWE-bench Pro approximates real repository work better than many classic isolated coding tasks, but it still does not measure all of software engineering. We know why withheld and private data can reduce contamination risk, and why even a newer benchmark can still contain flawed tasks. We also know that harnesses, tools, compute budgets, and even infrastructure can influence the result.

None of this removes the value of the benchmark. Quite the opposite: only once we understand its boundaries can we use the measured information sensibly.

Benchmarks are useful when we understand what they measure. They become dangerous when we infer something from a precise number that the benchmark never measured.

A benchmark is not a verdict on a model. It is a measuring instrument.

And almost as a side effect, we have learned something else: as soon as we evaluate a coding agent, we are rarely talking about the model alone. Context, tools, the harness, persistent state, and the way information is supplied all influence its work.

Before we later examine why the same model can produce different solutions, we first need to understand how those surrounding layers differ: Context, Memory, Skills, and Agents.

That is the subject of the next article.

Sources

[1] Qwen Team: Qwen3.8-27B Model Card. Hugging Face, 2026; including evaluation metadata for ScaleAI/SWE-bench_Pro published on August 14, 2026.

[2] Anthropic: Project Glasswing: Securing critical software for the AI era. 2026. Reported SWE-bench Pro score for Claude Opus 4.6: 53.4%.

[3] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R. Narasimhan: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024; supplemented by the official SWE-bench documentation and leaderboards, current as of September 2026.

[4] OpenAI: Why SWE-bench Verified no longer measures frontier coding capabilities. February 23, 2026.

[5] OpenAI: Introducing SWE-bench Verified. August 13, 2024.

[6] Scale AI Research Team: SWE-Bench Pro: Raising the Bar for Agentic Coding. September 19, 2025.

[7] OpenAI: Separating signal from noise in coding evaluations. July 8, 2026.

[8] Mark Chen et al.: Evaluating Large Language Models Trained on Code. 2021. arXiv:2107.03374.

[9] Gian Segato / Anthropic: Quantifying infrastructure noise in agentic coding evals. Anthropic Engineering, February 5, 2026.

[10] Terminal-Bench Team: Terminal-Bench 2.1, Release Notes, May 6, 2026; supplemented by Terminal-Bench 3.0 and the benchmark overview, retrieved September 10, 2026. Each version reference belongs to its specific evaluation.

[11] Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke: SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ICML 2025.

[12] Ibragim Badertdinov et al.: SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. 2025. arXiv:2505.20411; Ibragim Badertdinov et al.: SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale. 2026. arXiv:2602.23866; supplemented by the SWE-rebench leaderboard, token-consumption analysis, retrieved September 10, 2026. The dynamic leaderboard is not an immutable benchmark time series.

[13] Naman Jain et al.: LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. 2024. arXiv:2403.07974.

[14] John Yang, Carlos E. Jimenez, Alex L. Zhang et al.: SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? ICLR 2025; SWE-bench Multimodal v2, September 1, 2026.

[15] Kabir Khandpur, Kilian Lieret, Carlos E. Jimenez, Ofir Press, John Yang: SWE-bench Multilingual. 300 curated tasks from 42 repositories across nine programming languages.