The Difficulty of Creating Hard, Fair, and Valid Benchmarks

Frontier models are very smart, and breaking them is hard. Anyone who has spent the last year trying to build a benchmark that a frontier model fails has had the same experience. You write a task you are sure is difficult, run it once, and the model solves it. You harden the task, run it again, and the model solves it again. Eventually you find something it fails at, and the reaction is relief. The task is hard, ship it.

There is a question that should get asked at that moment and almost never is: did I find a real gap, or did I build a trap? The benchmark reports a low score, the low score is read as evidence that the model cannot do something, and nobody checks whether a model behaving perfectly could have passed, or whether passing would have meant anything.

This post is about that question and why it goes unasked. Creating a benchmark is a search for scenarios where models fail. The difficulty in that search is not finding a failure, but finding something that is hard, fair, and valid at the same time. The field tends to stop at the first failure it sees, and that is how it gets the other two wrong.

Reasoning is saturating

Breaking a frontier model is hard because the models are, at this point, extremely rational. That's not to say they are perfect reasoners - they still drop a case, lose a constraint over a long chain, or produce an argument that is subtly circular. But they are very, very, very good, and the set of problems where a frontier model reasons wrongly has shrunk to something incredibly small. Give them a problem whose answer can be derived from what they know and what they are told, and almost always they derive it correctly. Most of their failures come from wrong premises (from past training, or the environment itself) rather than wrong conclusions.

You can see the consequence in the domains where reasoning alone is enough. Mathematics is the cleanest case. Competition math went from a differentiator to a solved category in a couple of years, and the harder tiers that replaced it are following the same curve. Models now also contribute to research mathematics, proving lemmas, closing open problems, and checking proofs at a scale no human group can match. Serious people argue that mathematics as a science, the whole enterprise of proving new theorems, may be substantially automated soon. Whether or not that timeline holds, the direction is clear. This is what a field looks like when reasoning is enough and the models have the reasoning.

Every time we find a scenario where the failure comes down to "you have to think carefully about this," the frontier catches up within a model generation, and often within a release.

That does not mean there is nothing models cannot do. There is plenty, but what is left is rarely a reasoning gap, and even when it is, it is small, scattered, and hard to hit on purpose. A few other things are still left, and they are harder to measure.

The missing knowledge lives in the world

Consider what frontier models are still visibly bad at: operating a computer through its interface, navigating a UI they have never seen, driving a physical process, interacting with the real world. None of these are reasoning problems. No amount of reasoning tells you which menu hides the export button, what this instrument does when you send it that command, or how this system behaves once you start poking at it. Those are facts, and the model does not have them.

For a long time, the knowledge models lacked lived in text: facts, procedures, conventions, domain expertise. That knowledge was abstract, and abstract knowledge can be written down and trained on. The knowledge that is still missing is increasingly not abstract. It lives in the world, and the only way to get it is to act in the world and see what happens: run the reaction and measure what comes out, drive the instrument and read it back, click through the software and watch how it responds, probe the system and record how it fails. None of this knowledge exists in text. It has to be collected, one interaction at a time.

This becomes even clearer when we use AI for science. Science is reasoning on top of evidence, and in most fields the evidence comes from probing the real world. A model cannot do science without that evidence, and it cannot get the evidence without knowing how to probe: the instrument, the sample, the anomalous reading nobody wrote down. We keep building science benchmarks and finding that the models know the science. What they lack is the contact with reality that produces new science. That is not a reasoning problem, and no benchmark built from reasoning will find it.

This is especially relevant when we consider the challenge of Recursive Self-Improvement (RSI). Reasoning about how to improve an AI system can only get you so far, and at some point you will need to run experiments, i.e., probe the real world. If we accept that models are already capable of proposing ideas for self-improvement, the missing link is knowing how to operate hardware and run experiments efficiently. Continuing to build on top of reasoning will not get us there.

Capability limits are not reasoning gaps

By capability we mean how much context a model can hold at once, how fast it works, how cheaply it runs, how long it can keep going. These matter enormously in practice, and models are far from the ceiling on any of them. But they are orthogonal to intelligence, and that is why they get a section of their own: they are easy to confuse with reasoning and knowledge and they are not either.

A model that fails because the record was longer than its attention, or because it ran out of budget before finishing, has not shown you a reasoning gap or a knowledge gap. It has shown you a capacity limit. The same model with a bigger window, or a helper to fan out across, would have passed. Benchmarks routinely conflate the three, and a task that is only hard because it is long is measuring capability, and a low score on it says nothing about whether the model understood the problem.

Coding is a good example - as a reasoning problem, coding is mostly solved. Give a frontier model a well-specified function, a contained bug, or a self-contained module, and it writes correct code at a rate that would have been unthinkable three years ago. Yet models still fail in real codebases, often enough that "AI cannot really code" remains a common opinion.

Our hypothesis is that almost none of that failure is reasoning, most of it is capability. The model does not remember what it learned last session, it cannot fit the whole codebase in context, so it works from a partial picture, and the missing part is where the bug was. What remains after that is a small set of behavioral failures, of a kind we return to below: picking one side of an ambiguous design decision and committing to it without revisiting, forgetting or never accounting for an implementation quirk, not analyzing enough before acting. Strip those out and very little is left that looks like "cannot code." A coding benchmark that does not separate these is measuring a mixture, and its score does not tell you which one to fix.

Hardening without checking makes tasks unfair

This is the central argument in this post, but before we get to that we need to define what we mean by fairness. In the scope of model evaluation, we mean this:

A benchmark is fair if it rewards correct outcomes and only correct outcomes, and what counts as correct is unambiguous.

Fairness is a property of the grader, not of the model. It says nothing about which route the model should take - any behavior that reaches a correct outcome gets credit, and no behavior that misses it does. The task is fair when those two sets line up, whatever the behaviors inside them look like.

This is different from writing down the steps a good solution takes and grading the model on whether it took them. That is process grading, and in this field it is understood to be a mistake, for a reason that will be familiar by the end of this post: the model gets punished for reaching the answer in a way the author did not think of. Under our definition, the author never has to imagine the route at all. The only two questions are whether a model that reached a correct outcome could miss credit, and whether a model that missed it could get credit. If either can happen, the task is unfair.

That second clause, that correctness be unambiguous, is there because when correctness is a matter of the author's judgment, the reward is really for matching that judgment, and a model can learn to match it without learning the task. Unambiguous is also not the same as single. "The materials support no answer" can be the correct outcome, and so can "either of these two choices." What the clause rules out is a grader that could have gone the other way. Many of the benchmarks we look at later fail here: they reward one of two defensible answers, or a confident answer where none was supported.

One more point, about cheating. A model that cheats by hardcoding the outputs or reading the answer key has passed, and failing it looks like grading the route after all. However, when cheating happens, the run never reaches the correct outcome: if the objective is to write a function that generalizes across environments, a model that writes a function that only passes the task’s test suite has effectively failed. Cheating can then be seen as reaching a wrong outcome, and giving it credit is a task design mistake that is unfair by our definition.

With that in hand, consider what happens when an author keeps pushing on each of the three axes we discussed earlier, without checking against it.

Reasoning. The author refuses to accept that reasoning has saturated and keeps hardening. The model keeps solving the tasks, until one day it does not. The natural reading is that the task finally became hard. More often than not, it became unfair. The step that took the task from solved to failed was not a step up in difficulty but a step into one of the traps below, and from the outside the two look the same. Occasionally the hardening does land on one of the rare genuine reasoning gaps, but that is the exception, and the author usually cannot tell which case they are in.

Why unfair rather than hard? Because if the model reasons correctly and still fails, either it reached a correct outcome and was not rewarded, or what counted as correct was not settled in advance. Either one is what unfair means. There is very little left to punish on the reasoning axis, and the little that remains is hard to target deliberately. So when a benchmark punishes reasoning at scale, task after task, it has almost certainly not found a hundred genuine reasoning failures.

Knowledge. The author picks a domain and writes tasks that turn on domain knowledge. The model has the knowledge and passes. The author, still convinced the domain is where the difficulty lives, keeps pushing, and the move that eventually works is to take the knowledge away: strip the reference from the environment, hide the fact the answer turns on, assume a convention the model has no way to learn. Now the model fails. But it fails because the task became unanswerable from what it was given, not because it lacked anything a competent person would have had. The author set out to test knowledge the model already has and ended up testing whether it could guess.

Capability. This is the hardest one to notice, because for a while the hardening works. Make the record longer, the number of things to juggle larger, the budget tighter, and the model cracks under load. But the task has changed at some point - it is no longer "can the model do this," but "how much of this can the model do." That is thus an optimization problem, and optimization problems have a limit. Somewhere along the load axis the task crosses from hard to impossible for any behavior.

The crossover point is unknowable from inside the task. Either you know the task is possible at this load, in which case you have a solution in hand and the load was never the obstacle, so there was nothing to optimize. Or you have pushed past the point where you can say that, and you no longer know whether the model failed because it could not or because nobody could. In the first case you did not need the hardening. In the second you are measuring nothing, and the score cannot tell you which case you are in.

Whatever axis the author was pushing on, pushing without checking lands in the same place, and the score it produces looks the same from outside as the score a real failure would produce.

The unfairness comes in recurring forms. The list is not exhaustive, and we keep finding new variants, but when we audit hardened benchmarks the difficulty most often comes down to one of these four:

1. Withholding key information. The answer depends on a fact that is not in the environment and could not be inferred from it. The model that asks for it, or flags its absence, is graded the same as the model that guesses.
2. Grading on hidden requirements. The rubric wants the report in a specific format, a particular caveat mentioned, the second-best option ruled out explicitly, and none of this was asked for. Process grading is the same trap with the requirements written as steps: the model reached the answer but did not take the route the author had in mind. Either way it is punished for not reading the author's mind.
3. Demanding one arm of an ambiguous decision. Two competent experts would disagree. The rubric picked one and scores the other as wrong. Nothing about the model's behavior distinguishes the two outcomes.
4. Making the task impossible. The clean version got solved, so the author added constraints until nothing satisfies them. Every model scores zero, the benchmark is "hard," and it measures nothing. Capability hardening pushed past the limit ends up here too.

The fourth trap has a partial version that is subtler and more common. The scale says the maximum is 100, but no behavior can score above 60, because some of the points are attached to things nobody could have done. The gap between the real ceiling and the stated one gets read as a capability gap, when it is a property of the benchmark.

Each of these produces a low score, but none of them produces a signal about what the model should have done differently. That is the point of the fairness definition: a benchmark that cannot say what a correct outcome would have been, or says it only after seeing the answer, cannot reward it, and so cannot train it.

Experts are not a fairness check

The reflexive answer to all of this is to bring in domain experts. Experts are necessary, and this section is not an argument against them. It is an argument against two things people conclude once they have them: that the tasks are therefore fair, and that the fairness problem has therefore been handled at scale.

Experts have domain knowledge, but domain knowledge is not knowledge about fairness in the sense above, and most experts have never needed the second kind. A physician can tell you whether the diagnosis is right. They cannot, without training in this specific problem, tell you whether the task gave the agent enough to reach that diagnosis, whether their rubric contains requirements they never stated, or whether the alternative they marked wrong is one their colleague down the hall would have chosen.

There is a structural reason for this, and it goes back to the fairness definition. An expert is the right person to say what the correct outcome is, or if a single trajectory meets certain criteria. An expert is not the right person to enumerate every way of reaching it. Fairness needs the second thing: the set of behaviors that get credit has to match the set of behaviors that get there, and the expert, who got there one way, will tend to write a rubric around that one way.

So expert-authored tasks fall into the same traps as everyone else's, and with more confidence, because the expert knows the answer and does not notice that it is not derivable from the materials, or that it is derivable by a route they did not take. "An expert wrote it" gets treated as evidence that a task is fair. It is evidence that the domain content is right, which is a different thing. Fairness is a separate question, and credentials on the author do not answer it.

Fair benchmarks can still measure the wrong thing

Suppose you do the work. You find scenarios where frontier models fail, you audit every one of them for unfairness, and they all pass. There is still a third question, and it gets skipped even by people who asked the first two: is it valid? Does the benchmark measure what you say it measures?

Validity only comes into play once you know the benchmark is hard and fair. Before that it is invalid by definition. An easy benchmark measures nothing about the frontier, because the frontier saturates it. An unfair benchmark measures nothing about the model, because the score reflects the author's traps rather than the model's behavior. Neither can support a claim about anything, so there is no validity question to ask. You earn the right to ask it by clearing the first two bars.

Even once you have earned it, the answer is often no. It is easy to find failures that mean nothing: Build a "science" benchmark where the agent has to run a real experiment, and reporting the result means logging it into the lab's data system through a fussy, idiosyncratic spreadsheet template where one misplaced column fails the task. Frontier models will trip on it. The failure is real, and it is fair: a model that learned the template's quirks would pass. But the difficulty is in the data entry, not in the science, and a low score tells you the model cannot yet wrangle a clunky form. It tells you nothing about its scientific reasoning, and if you publish it as a science benchmark, every conclusion drawn from it is wrong.

Someone will object that data entry is part of doing science, and it is. But so is any number of skills that surround the science without being it. Language, for one: reading the protocol is part of science, writing up the result is part of science, and a benchmark whose difficulty was really the density of the English in the protocol would be measuring reading comprehension, however true it is that scientists read. Being part of science does not make a benchmark about it a valid measure of science. Validity turns on what the difficulty is made of, and here it is made of the form.

Capability benchmarks have their own version of this, separate from the impossibility limit above. Below the point where the task becomes impossible, a capability benchmark is still fair: the model could have done it and did not. But before that point there is a second threshold, where the load being measured stops corresponding to anything a practical use would demand. A benchmark that asks the model to hold an entire enterprise codebase in one context, cross-reference ten thousand records without a tool, or finish in a tenth of the time any workflow needs, may be fair and meaningless at the same time. Nobody needs that, so nothing is learned from the model not having it. The optimization has to stop where the capacity stops being useful, not only where the task stops being possible, and the first of those usually arrives sooner.

Point is, hardness and fairness are independent of validity; you can get the first two right and still ship a benchmark whose headline number means nothing. Put the two failure routes together and a large share of the benchmarks out there are invalid, more than we think most people would guess. Some because they are unfair, and the low score is the author's doing. Others because they are fair but say nothing about the thing in the title: the coding benchmark that measures harness familiarity, the science benchmark that measures hardware usage. Either way, the number on the leaderboard does not support the sentence written next to it.

One fair task does not make a benchmark

As if this problem wasn't already hard enough, there are still more obstacles to clear.

Suppose you passed all of the three criteria: hardness, fairness, and validity. You have a scenario where a frontier model fails, the failure is fair, and it means what you say it means. Congratulations, you have one data point! But a benchmark needs many, and now you have to find a way to reach that many.

The obvious move is to generate more data points synthetically. Take the scenario, change the domain, change the names, change the numbers, and produce a hundred variants. The trouble is that a genuine failure mode is usually a unique scenario - what makes the model fail is a specific configuration: this piece of evidence contradicting that one, this source looking more authoritative than it is, this decision point arriving before the confirming fact does. Swapping the hospital for a law firm and the lab result for a contract clause keeps the configuration and changes the skin. The model that fails the original fails the variant for the same reason. Changing the skin may buy some diversity, and it may be worth doing, but how much of a new data point the variant is, and how much a model learns from the second skin that it did not learn from the first, are open questions on their own, and the honest default is to count it as one failure mode, not two.

Real variety means a different configuration, which means a different failure mode, which means the whole search starts again: find it, check it is fair, check it is valid. Slow authoring is the smaller problem. The larger one is that every hard, fair, valid task has to be found before it can be written, and you cannot schedule how many you will find.

This describes a lot of the benchmarks that already exist. Look closely at a large benchmark with hundreds of tasks across a dozen domains and you often find a handful of failure modes wearing different skins. The tasks look diverse because the domains are diverse. The reasons the model fails are not.

This is easy to miss while you are building the benchmark, and missing it is not a sign of carelessness. From inside the project, it looks like the model is failing in radiology and in tax law and in materials science, and the natural reading is that each is a separate knowledge gap. It takes stepping back from the domains and looking at the shape of each failure to see that they are the same three or four shapes repeated.

The same holds for benchmarks authored by experts or mined from natural sources, which feel more right. The tasks were not generated. They were written by someone who does the job, or pulled from real tickets and real papers, and that provenance reads as diversity. But provenance is not variety.

There are enormously many ways real work goes wrong, and an expert can list a great many of them. The catch is that this work has already been done. Years of expert-written tasks and mined corpora have taught the models nearly all of it, and the frontier no longer fails on the ways things usually go wrong. What it still fails on is a small handful of the remaining ones, and an expert writing fifty tasks, or a corpus of a thousand tickets, hits that handful over and over, dressed in whatever the domain happens to be. Expert-authored, mined, and generated benchmarks differ in how they look, but in how many distinct things they still measure at the frontier, they are not that different.

If the benchmarks are broken, why do models keep passing them?

There is an obvious objection to all of this. If so many benchmarks are unfair or invalid, they should be mostly noise, and noise does not trend. Yet scores on these benchmarks climb, release after release.

We think two things are going on at once, and the open question is the proportion between them.

The first we have already argued for. Even a mostly broken benchmark carries signal, just very sparse signal: the small handful of real failure modes that survive the hardening process. Models learn from those, and when they do they climb on the whole benchmark, because the same handful is what the rest of the tasks were re-skinning. Whatever share of the climb comes from this is real progress, just far noisier and more expensive than it needed to be.

The second is that models are also being taught the wrong behavior. A benchmark that rewards one arm of an ambiguous decision teaches the model to pick that arm. A benchmark whose difficulty is a hidden requirement teaches the model to guess what this kind of author wants. A benchmark that withholds a key fact and grades the answer anyway teaches the model that a plausible guess is worth more than an honest "I cannot tell from this." Train against enough of these and the score goes up, not because the model got better at the task, but because it got better at the benchmark.

This is Goodhart's law: a measure that becomes a target stops being a measure. "Benchmaxxing" is the field's word for doing it on purpose. What we are describing is the same thing happening by accident, one unfair task at a time. Whatever share of the climb comes from this is not progress. It is a model learning to behave the way a badly built benchmark wants, which is worse than no training at all.

It would also explain something many people have noticed: models keep getting dramatically better on benchmarks, and the improvement does not show up in day-to-day use at anything like the same rate. If a good part of the climb is the model learning what benchmarks reward, that is what you would expect.

At some point this becomes dangerous rather than merely disappointing. The behaviors an unfair benchmark rewards, guessing instead of checking, committing instead of asking, answering where the honest response is that the materials do not support an answer, are the opposite of what a careful reasoner does, so every gradient step toward the benchmark is a step away from the judgment that made the model good in the first place. We are teaching it to be less intelligent, and to fail in the hard-to-catch way. Earlier generations, when they were wrong, were blatantly wrong, and a blatant error gets caught. A model shaped by unfair benchmarks fails with a confident answer and a coherent justification, and that failure gets through review. We may be training the errors to be harder to see rather than training them away.

Both forces push the score in the same direction, which is why the score alone cannot separate them. That is itself an argument for checking. A benchmark you have checked for fairness and validity is one where a rising score means the good thing. A benchmark you have not checked is one where a rising score is some mixture of the two, and you will not know the mix until the model starts to feel worse.

What is left, and what comes next

So where does this leave us? Reasoning has saturated, and most of what gets hardened into a benchmark turns out void. If both of those are true, the useful question is no longer how to make a task harder but where the real gaps are, the ones worth the cost of building a benchmark around.

The benchmarks that drive the field, the famous ones that set the headlines, all live in the same enclosed space, and so do all the ones this post has been describing. A single agent is given everything it will get up front, does the work by thinking about it, and hands back a clear, verifiable answer. Benchmarks exist outside that space, but they are not the ones the field steers by. The space is bounded, and the field has spent years filling it. Any gap simple enough to write into a task has been written into one, and the models have closed most of them.

What is still open in the bounded space

We think the largest gap still open inside that space is behavior. Give a frontier model a task with every fact it needs and it will almost always reason correctly. Take some of the facts away, or make them disagree with each other, and the model starts doing something other than reasoning badly. It stops knowing when to trust a source, what to trust in it, or who to trust when sources disagree. It takes a number from a cached table because the table was there, when the raw data would have told a different story. It accepts a document's claim about itself. It leaps from "plausible" to "established" with nothing to stand on, then builds correctly on the wrong foundation. It commits early and reasons its way into justifying the commitment.

This looks like bad inference, but it is bad judgment about when to infer at all, and about what a careful person checks first. How large that gap is, though, we do not know.

But with the closing of this space, a new one is opening up. Now that models are intelligent enough to reason through most tough inference problems, we have a whole new set of environments and tasks we can test them in.

That new space has no bound in sight, and almost none of it has been touched. We think its defining feature is that reasoning is not what makes it hard - the models can already reason well enough. What they will fail at is something else, and we do not yet know everything that something else includes. What follows are the directions we find most interesting.

Knowledge that has to be fetched from the world

The gap closest to what we have now is the one we covered at the start: knowledge that only exists once someone has done the thing and watched what happened, and that a model cannot reason its way to. The only way to get it is to interact with the world and see.

What makes this direction hard to build for is not grading. It is that the fact has to be collected from the world before it can go into a task, and collecting it means building the interface, wiring up the instrument, or running the experiment. Every task is an act of fieldwork, and fieldwork is slow, expensive, and does not parallelize. That is why knowledge benchmarks are rare relative to how much of the gap they cover, and why nobody has found a way to build them faster than one act of fieldwork per task.

Another difficulty is that once the knowledge has been collected and written down, it stops being this kind of knowledge. It is text now, it can be trained on, and the next model will simply have it. A benchmark built on world-contact knowledge falls back into the bounded space the moment its answers reach the internet, and the fieldwork has to be done again, on something new. Nothing about this makes the direction less worth pursuing, but it does mean that no benchmark of this kind stays hard for long, and that the gap it measures is not so much a fixed body of missing facts as the frontier's distance from the world at any given moment.

Ambiguity and taste

The next gap is ambiguity, which is where the behavioral failures above were pointing the whole time. In the bounded space, the facts were incomplete or in conflict but an answer still existed. Out here, often it does not. The information does not settle the matter, the sources disagree and none of them is authoritative, and there is no rationally correct answer to reason toward. Models can handle problems with a derivable answer, but they are much worse at the ones where a competent person would say "it depends," pick a course anyway, and be right to. We think that skill, acting well when nothing settles the question, is the largest gap that remains.

It is also the thing we are least able to grade. Everything in this post about fairness rests on there being an unambiguously correct outcome to reward, and ambiguity is where that clause fails. If two competent experts would act differently, a benchmark that rewards one of them has committed the third trap above, and one that rewards both has stopped discriminating. That the thing models are worst at is the thing we can least grade is no coincidence. We cannot grade it, so we cannot reward it, so we have not found a way to train it, and the models stay weak there. How to grade judgment under ambiguity without either picking a side or refusing to discriminate is an open problem.

Taste, aesthetic judgment included, has the same shape and, so far, lower stakes. A model produces output that is correct and reads badly, or looks wrong, or is not what a person with judgment would have made. While the output is a paragraph this is easy to let slide, and almost nobody benchmarks it, because there is no agreed way to grade it. Once the output is a design, an interface, or anything a person has to look at and live with, it stops being something you can let slide. And as with ambiguity, correct inference does not produce it and two good judges will not always agree on it.

Meta-Work

The last direction we will discuss is work about work. Almost everything the field measures today is a single task, done once, by a single agent, and judged on its output. Real work is rarely shaped like that. It is a project made of many tasks, run over a horizon long enough that the model has to manage its own history instead of being handed it, often alongside other agents whose knowledge and goals differ from its own. Call it meta-work: deciding what to do next, what to delegate, what to revisit, and what to trust from the agents around you.

What changes here is the unit of evaluation. The thing being judged is no longer an answer but a campaign. The failures that matter are the ones that surface after the hundredth step, or in how two agents deal with each other, rather than in what any one of them produced alone. A model can be excellent at every individual task and still run the project badly, because it made greedy decisions, it committed early to a plan, lost track of a decision it made earlier, or took another agent's confident report at face value. These are the behavioral failures from the bounded space again, only now they compound over time and across agents.

Meta-work also means constantly starting over in unfamiliar territory. Give a model a new set of tools and conventions and it takes longer than a person would to find its footing, and sometimes never does. A model tuned for one kind of work is often mediocre at the next one over, in ways that do not track how hard the second kind is. Benchmarks are blind to both, because each fixes a harness and measures inside it, and each is a single use case, so the cost of arriving never lands in any score. In work made of many machines and many kinds of tasks, that cost is most of the job.

We do not expect any of this to require new reasoning. The reasoning and the text knowledge models already have should be enough. What we expect instead is a set of failures we have only started to see: reasoning over more dimensions than a single transcript holds, deciding under uncertainty when the uncertainty is about the world rather than the text, and coordinating with agents that may be wrong. None of those is one more step of inference, and benchmarks that catch them are still few.

Running through all of this is capability. Whatever space a task lives in, the model doing it can still run out of context, forget across sessions, take too long, or cost too much, and these limits bind on real work more often than any reasoning gap does. They are not a gap in either space so much as a ceiling over both, and it is a ceiling that moves with each model rather than one that gets reached. There is still a lot of room to push on it, within the two limits from earlier.

What this means in practice

For the last few years the dominant recipe has been to take real-world scenarios, throw many of them at the model, and let it learn both the reasoning and the knowledge from the data. That worked for a specific reason: there was a lot to learn. Models lacked both the knowledge and the reasoning, so nearly any hard task carried signal.

That is no longer the situation, and the recipe has inverted. The current pattern is to generate a very large number of tasks and then adversarially harden them until the frontier fails. We have run this process ourselves, many times, and this is what happens to the tasks:

• Most of the hardened tasks end up unfair. They land in one of the traps above, or in one we have not catalogued yet. The score is low, and the score is noise.
• Some land on real failures. A genuine knowledge gap, or a real behavioral failure. These are the ones worth keeping.
• The ratio is bad. Depending on the domain, the keepers are a small minority of what gets built.

This would be fine if tasks were cheap, but they are not. A single realistic, gradeable, multi-step task in medicine, law, or experimental science costs real expert time and real engineering time, and verifying that it is fair costs more than authoring it. Building many to throw most away is not a strategy. It is an expensive way to find a handful of tasks you could have aimed at directly.

That leads to two rules, one for people building benchmarks and one for people reading them.

If you are building a benchmark, aim directly. Instead of mass-producing tasks and filtering for the ones that fail, go looking for a weakness first, confirm it is real, and then build the task around it. Of the gaps in the previous section, three can be turned into fair, gradable tasks today:

1. Where does its behavior fail? When, what, and who it trusts, and the leaps it makes without grounds. These are the last reasoning-adjacent gaps, they are gradeable, and a real one is a rare data point. Be honest about how many distinct ones you have once the skins are stripped off.
2. Where does it lack knowledge? Not abstract knowledge, which it mostly has, but the world-contact kind: interfaces, instruments, institutional quirks, how a specific system fails. Expect fieldwork.
3. Where does it run out of capability? Context, memory across sessions, speed, cost. These are real limits and worth measuring, on two conditions. Label the benchmark as a capability benchmark rather than dressing it up as intelligence, and stop hardening where you can still show the task is possible and still name a use that needs it.

Beyond those sit ambiguity, taste, and meta-work. Nobody knows how to grade them well yet, and the first benchmarks that do will be hard to build. But the space is almost empty, the gaps are wide, and anything that measures them fairly will be worth far more than one more task in the bounded space. Go after them if you can!

Either way, what is left to measure is narrow, awkward, and expensive to build for, which is exactly why the field keeps drifting back to tasks that are none of those things and calling them hard. That the tasks must be hard, fair, and valid does not change. Where the hard, fair, and valid ones are still waiting to be found does.

If you are reading a benchmark, ask three questions. Is it hard? Is it fair? Is it valid? The first is given, since nobody releases a benchmark the current model aces. The second is where the damage happens, because unfair is the cheapest way to make a score go down and an unfair benchmark trains the wrong thing. The third is the one that gets skipped even when the first two were asked, because a hard, fair benchmark can still measure something other than what its title says. If the answer to either of the last two is no, the conclusion drawn from the number is void.

Conclusion

Frontier models are good enough now that a task which breaks one is no longer evidence of anything on its own. It might be a real gap. It might be a fact the author withheld, a requirement the author never stated, a side the author picked in a call that had two defensible answers, or a load nobody could have carried. From the outside, all of these look the same: a low score and a hard-sounding name. Telling them apart takes the two checks this post has been about, most benchmarks never get either, and the field has spent a great deal of money learning very little as a result.

The interesting work has not been done yet. Inside the space we have been working, there is still behavior to measure, and it can be measured fairly. Beyond it there is knowledge that has to be fetched from the world, judgment under ambiguity, taste, and work about work, most of it untouched and none of it reachable by hardening a text task one more time. Finding scenarios where models fail was never the problem. Finding ones that mean something is.