Introduction: how this benchmark works

In short: a modern AI agent is usually a pipeline of blocks running in a loop. It is only as strong as its weakest block, because loops tend to accumulate errors until they fail. A bad classification at step 3 is a wrong refund at step 12. We took the best agentic architectures from BitGN, traced where they snapped, and reproduced those conditions across modern LLMs. This report tells you which models hold at the points where good architectures break, and what that costs you in money and latency. Reports are linked below.

This LLM benchmark continues a line of research into the practical adoption of LLMs and AI in business workloads that began in July 2023. Back then, the community was taking its first steps in building systems with LLMs under the hood, and the questions were: “Which LLM model should I pick for my BI assistant?” and “Which LLM is best for on-premises OCR of manufacturing data sheets?”

Our first generation of benchmarks answered these questions. It consisted of a collection of evals extracted from business workloads, and we used them to test different models. EU and US companies rely on these benchmarks to quickly assess models for new projects.

However, progress moves fast these days. Over the following year and a half, modern LLMs improved so much (alongside our understanding of how to design and deploy systems with them) that the first generation of the benchmark became saturated. New models were getting near-perfect scores. The benchmark therefore had to be redesigned to include more demanding challenges and new patterns for getting higher-quality results from models (e.g. Schema-Guided Reasoning). The new version came out in January 2025.

Progress did not stop there, nor did learning and research in the AI community. As we’ve seen in recent cases of AI adoption, the question is no longer “What is the best LLM model?” (all frontier models are more than capable), but rather “What is the best LLM model for an AI agent to do the job quickly, cheaply, and securely?”

Properly answering that question requires a new line of research:

1. What do the best AI agents look like?
2. What are the common building patterns of these agents?
3. Which models work best in supporting the best agentic architectures?

To handle this properly, we created BitGN.com - a platform that captures business workloads and uses them to objectively verify the behavior of different agentic architectures. We ran two public challenges, Personal and Trustworthy Agents (https://bitgn.com/challenge/PAC) and Agentic Commerce (https://bitgn.com/challenge/ecom), attracting more than 1000 engineers across 100+ locations. More importantly, we captured 100M agentic interactions with our system across 2.4M trials. Agentic architectures were ranked on leaderboards across different performance criteria; teams contributed architectural deep dives and revealed secrets about their agents.

An interaction is a single tool call from an agent to the BitGN runtime; a trial is one agent run against one task instance.

That answered questions 1 and 2. The best AI agents are lean: specialized, domain-specific tools (created at runtime or during training) coupled with strategically placed checks and guardrails. We had the architectures, the source code, the deep dives, and a dataset of how they actually performed.

You can read about these agents in BitGN Insights and browse the architecture write-ups currently in review in our PR queue.

Question 3 remained, and it actually matters the most today, when AI agents need to go through long trajectories to solve complex tasks.

An agent is not a model. It is a set of blocks (a preflight router, a policy check, a tool writer, an analyst, a verifier) wired into a loop. The loop allows agents to plan and adapt, but also makes everything fragile in a specific way: each block hands its output, its errors, and its accumulated context to the next one. Errors compound. The agent is as strong as its weakest block, and the weakest block is usually not where you expect it.

So we did not ask “which model is smartest.” We took the 2.4M trials, traced the trajectories of the best-performing harnesses, and looked for the snapping points - conditions under which strong architectures degrade, lose the thread, or get talked into something. Then we reproduced those conditions across modern models.

It is now July 2026, exactly a year and a half since the last major benchmark redesign. It is time for the Agentic LLM Benchmarks. Here are the results.

How is this leaderboard different from the BitGN leaderboards?

BitGN ranks whole agent architectures; this report evaluates LLMs using patterns extracted from the strongest architectures.

How is this benchmark different from other benchmarks?

This is a closed-source benchmark that measures LLM performance under the constraints of business workloads, currently focusing on agentic commerce. It uses patterns drawn from top-performing agentic harnesses, as measured in blind-submission challenges hosted on the BitGN platform.

Compared to the benchmarks you already know:

• SWE-bench, Terminal-Bench and similar measure a model inside a coding harness. Here, code is a means and not the goal - it shows up as a tool the agent writes to solve a business task.
• τ-bench / τ²-bench also evaluate agents in retail-like domains with deterministic checks. The difference is provenance and scope: our tasks and fault scenarios are not authored by hand - they are extracted from the ways that hundreds of competing real architectures actually failed, and they run against a snapshot of a commerce environment (warehouse, customer records, policy book), with the agentic trajectory preserved and with adversarial pressure built in.
• MMLU, GAIA and friends measure general capability. We measure capability under cost, latency, and hostile pressure - the constraints that decide whether an agent ships.
• Public benchmarks eventually leak into training data. This one is private, and its tasks are derived from workloads captured after the fact.

What kinds of tasks are there in the benchmark?

The benchmark is built from the patterns that top BitGN harnesses actually use (read more here). For example: routing and classification, security checks, tool generation, dynamic analysis through AI coding. However, we don’t test these patterns in the happy path. We push them to their limits and test them where they break.

For each pattern, we went into the BitGN dataset and found the bottlenecks and fault scenarios: the inputs that made good routers misclassify, the policy phrasings that good checkers waved through, the CSV shapes that made good tool-writers emit code that compiled and lied. When multiple top-performing architectures failed in the same way, we captured the root cause and turned it into a repeatable eval.

Each eval is a snapshot of a real trajectory, taken at the moment it snapped: the accumulated context, the prior errors, the state the agent had talked itself into. That step is then replayed against every model. The pattern still has to do its full job. We just make it do that job from exactly where the best architectures failed.

This is also why the benchmark should not saturate the way the first generation did. Its tasks are mined from the strain points in agentic trajectories. As architectures improve, they stop failing in the old ways and start failing in new ones - and those become the next evals.

Here are a few examples of actual tasks:

1. Given potentially malicious input, classify e-commerce assistant tasks for the agent’s preflight router.
2. Given accumulated context, write a specialized Python tool that takes a CSV file in this layout and performs availability calculations for all branches according to these rules.
3. Given long context and a few failures, write SQL to list all retriable 3DS transactions according to these policies and rules.

How do we run the models?

Models are served through OpenRouter, which routes each request to the best available provider. The exception is an accelerated run, where a model is deliberately pinned to a hardware accelerator (e.g. Cerebras Wafer-Scale Engine or Groq LPU); those rows carry a lightning icon.

Two consequences to keep in mind while reading the charts:

• Speed and cost are properties of the model and its serving provider. In practice, providers compete on the same open market (nobody wins by being slower and pricier), so OpenRouter routing and market pressure keep the spread narrow. Some variance remains, but it is not too big: the model dominates.
• Reasoning effort is a first-class variable. Where it applies, it is listed in the table, and several models appear at more than one level (e.g. low and high) - because effort alone can move a model across the Pareto front.

Costs are based on the list API prices at the time of the run (plus OpenRouter markup), and the run date is shown in the benchmark. Treat Cost as a relative indicator between models rather than a quote.

How do we evaluate tasks?

The backend evaluation logic is similar to the automated, deterministic verification used within the BitGN platform.

In each case, we have a precise, deterministic way to measure the quality of the outcome. We avoid the inherent problems of the LLM-as-a-judge approach by completely avoiding LLMs in the grading pipeline.

Instead, each task gets its own grading code that scores model outputs according to predefined rules. Each model starts each task with a score of 1.0 and is penalized for each detected problem.

For example, in a code-generation task, the model can receive a score of 1.0 for a perfect answer, determined by compiling the code and then running it through a set of predefined unit tests. Otherwise:

• If the code has syntax errors and does not run, the score is reduced to 0.0.
• If the code runs but unit tests fail, we deduct 0.3 for each failing unit test, down to a floor of 0.1 (the credit for producing code that at least compiles).

Penalty weights are calibrated per task: a task with two unit tests and a task with eight do not carry the same deduction per test. Task scores are then averaged into the final score. Sub-scores such as AI Code are averages over a subset of tasks.

Deterministic grading comes with a deliberate trade-off: it can only measure what can be measured deterministically. We consider that a feature, not a limitation. It is the same rule we apply to our own engineering - we build only what can be measured deterministically, because that is where quality can be tracked and improved (Quality is a trajectory). If a task cannot be measured, it cannot be engineered. It is voodoo, and it has no place in a business-critical system.

How to read this benchmark

You can start reading the benchmark by looking at the best-performing model, then looking at its cost and speed and realizing that it is probably too expensive and slow to make sense for most projects.

For instance, at the time of writing, the top-ranked model is GPT-5.5 Pro, with a median pattern speed of about 1:40 (1 minute and 40 seconds per request) and a benchmark workload cost of about EUR 230.

This is the best-scoring model, but also the slowest and most expensive. You may therefore need something slightly less accurate, but much cheaper and faster. Welcome to the Pareto frontier!

The Pareto front represents the set of optimal solutions where you cannot improve one objective without worsening another. It eliminates suboptimal choices by finding the perfect balance (or trade-off) between competing goals.

You then need to pick what matters most to you - speed or cost, while retaining acceptable accuracy. That will depend on the type of project. For instance, interactive real-time AI agents need fast responses, while background batch AI jobs can run more slowly but must be cost-efficient to handle large workloads.

At this point, you only need to pay attention to the charts (“Final score vs workload cost” and “Final score vs pattern speed”) and pick models from the Pareto front that correspond to the balance between accuracy, speed, and cost that you find acceptable in your project.

Some models will even be present on both charts!

While looking at potentially interesting models, you can also pay attention to a few extra indicators:

• The Breach column shows the number of security breaches, prompt injections, and hijack attempts that we were able to sneak past the model, out of all attempts embedded in the workload. The fewer breaches, the more resilient the model is. In agentic commerce this is an important metric. Things like hostile checkouts, ranking manipulation by merchants, and agents being talked into moving money they shouldn’t are the core risks of letting agents touch commerce infrastructure.
• The Fatal column shows the number of requests that were not served at all: the model refused to answer, the constrained decoding engine failed, or the inference provider reported another problem. We do not separate them. In production, an unserved request is an unserved request. Since OpenRouter already routes to the provider with the best uptime, a fatal is a fatal.
• Speed is the median duration (minutes:seconds) of requests to this model within the workload. Lower is obviously better.
• Cost is the benchmark workload cost, scaled to a reference production workload. It gives you an idea of how expensive the model is compared with the others.
• AI Code is the sub-score for tasks involving code generation, verification, or analysis.
• If a model has open weights (meaning it can be downloaded and run on your own hardware), it will have a download icon.
• If a model was run on a provider’s hardware accelerator (e.g. Cerebras Wafer-Scale Engine or Groq LPU), it will have a lightning icon.

One caveat before you act on the rankings. These results describe how models behave at the stress points of these patterns (the ones that won on BitGN, and the ones we see in production across startups and enterprises in the EU and the USA). That makes them a strong default. But a model that survives someone else’s weakest link is not automatically a model that survives yours. If your harness is fundamentally different, you’ll need to find your own snapping points. To do so - follow the approach we’ve outlined earlier.

What this benchmark does not measure

• Whole agent architectures. That is what the BitGN leaderboards are for.
• Multimodal capability. Everything here is text and code.
• Non-English workloads (although our multilingual tests show that the gap between languages is getting narrower with each passing year).
• Time-to-first-token or streaming behavior. Speed is end-to-end request duration.
• Chat quality, creativity, or anything that would require a human or an LLM to judge. If we cannot grade it deterministically, it is not in here.

Can I run this benchmark myself?

No, and that is deliberate. A public benchmark eventually leaks into training data and stops measuring anything. We keep it closed on purpose.

The platform that produced the data for this benchmark is open.

BitGN accepts any agent of any architecture. You can run yours against the simulated runtime and get detailed grading explanations back: what your agent did, where it broke, and why. That is the same feedback loop the top teams used.

You can see where you land against them on the leaderboards, and read their published architectures if you want to borrow from hundreds of diverse architectures that have already been ranked against each other.

From there you have two options: steer toward the patterns that win, or stay off that path deliberately and gather your own agentic trajectories. In the latter case you can mine your own failure points and build the benchmark that fits your workload.

That second path is exactly what we did. Nothing about it requires our task set.