Vetto Science Bench

Can AI do science?

Every few months another AI system is presented as able to do science, whether as a research collaborator, an autonomous lab assistant or an AI scientist that runs experiments and analyses from start to finish. These are bold claims, and if they hold up, they will change how research is done. However, if they do not, labs that rely on these systems will publish confident results that turn out to be wrong.

The trouble is that telling whether the outcome of AI-powered research is correct is extremely hard. Many science benchmarks test what a model knows, through exam-style questions, recall of established facts or the reproduction of a published result, and a model can do well on all of them simply by knowing what the textbook says. Open-ended research, on the other hand, is hard to grade, because nobody knows the right answer in advance and a wrong result can take years to come to light.

There are still parts of research that are verifiable, though. For example, much of the day-to-day work in research is data analysis, and data analysis can be checked. And while it is true that frontier models are already very good at writing code, they still lack the judgment that good research depends on, such as deciding which method should be applied to a specific problem. That judgment is what we should be aiming models at.

Vetto Science Bench is designed to measure exactly these skills, which coding benchmarks do not capture. Each task puts an agent in a terminal with realistic data and a research question. To solve it, the agent has to explore the data, choose a method, check whether that method's assumptions hold, and deliver an answer or a working analysis pipeline, which we grade against held-out ground truth. The tasks are built so that the textbook approach alone is not enough: the data has features that a careful researcher in the field would catch, such as an instrument artifact, a selection effect or a hidden interaction, and the analysis only succeeds if it accounts for them.

The benchmark consists of 50 such tasks from 14 disciplines, ranging from MRI physics and seismology to population genetics and battery chemistry, and they draw on a survey of more than 100 scientists and researchers about where AI falls short in their day-to-day work. We ran 13 frontier models on it, and even the best one failed more than half of the time. Most failed more than 80% of the time, and five of the tasks were never solved by any model.

The tasks are spread widely across the 14 disciplines. Most come from empirical sciences such as geoscience, biology, medicine and physics, where analysing data is a large part of the daily work, and only a few are pure statistics or mathematics.

In the rest of this post we describe the tasks, walk through one of them, look at why models fail, and explain how we made sure that the tasks are fair. Detailed results on cost, trial length and per-task performance are in the appendix.

The task suite

Each task is a self-contained environment in the same format as Terminal-Bench Science. The agent gets a set of data files, a short instruction, and a terminal with Python and the usual scientific libraries. Depending on the task, it submits either a final answer or a solver that we run on held-out cases, and a hidden grader scores the result on a continuous scale, such as an error, an R² or a ranking score.

Because the grades are continuous, each task has a correct range rather than a single right answer. The range has one edge, a pass threshold, and any result beyond it passes, with no ceiling on how good a result can be. We set the threshold by having two solvers who already know the right method attempt the task, and we place it with a margin between their scores and the best plausible but wrong approach, which must still fall outside the range.

An example task

To make this concrete, here is one of the tasks in the benchmark: a medical imaging problem that every model we tested got wrong in the same way. The task builds on the fact that MRI scanners can measure how much of a tissue is fat, which doctors use to track conditions such as fatty liver disease. In this task, the agent gets raw scans of test objects with known amounts of oil mixed in, and has to write a program that reports the fat percentage at every point of a new scan.

What makes the task difficult is the kind of fat in the test objects. The standard method assumes that the fat looks like human liver fat, but here the oil is a different kind, and part of its signal hides inside the water signal. The agent is not told what the samples are, so a careful analyst estimates the fat spectrum from the data instead of assuming one. The standard method still runs without errors and produces sensible-looking maps, but its numbers come out several percentage points off. In our first test runs, every model we tried took that standard route and missed by two to three times the allowed error. Across the full benchmark runs, only GPT-6.1 Sol solved the task, once in three tries.

Why models fail

Most failures on Vetto Science Bench did not come from broken code or from missing scientific knowledge, but from how the models behaved during the analysis. In this section we look at what kinds of problems the tasks contain and at how the models handled them.

We classified the tasks into five types according to the kind of difficulty they contain. We did not aim for these difficulties when creating the tasks. Each one came about naturally as our task-creation pipeline searched for problems that models get wrong, so they reflect real gaps in how models analyse data. Each difficulty is described below, along with an example from the benchmark and the number of tasks that contain it.

Although these difficulties are hard to spot, we make sure that every task can still be solved by someone who looks carefully. The evidence of the problem is always somewhere in the data the agent receives, and handling it only requires methods that a researcher in the field would know or that the instruction explains.

We also classified every failed trial on the benchmark, 1,721 in all, by the main reason it failed, using its full trajectory together with the task's design notes. We found four recurring failure modes. A model can never check for the problem (F1), applying the textbook method without running the check that would reveal it. It can see the problem and dismiss it (F2), treating a number in its own output that points to the problem as noise. It can see the problem and handle it wrong (F3), correcting for it with the wrong model. Or it can validate where the problem is invisible (F4), testing its method on a part of the data that cannot expose the difficulty. Other failures, such as near misses and modelling errors unrelated to the task's difficulty, were rare, about one in twenty, and we group them together. The figure below breaks the failures down by the kind of difficulty and by model.

Across all 14 configurations, 95% of failures came from the task's difficulty rather than from unrelated mistakes. The two most common failure modes, dismissing the problem (F2, 601 trials) and handling it the wrong way (F3, 466 trials), can only happen after the model has seen the problem, for example in a number it printed or a check it ran. Together they make up 65% of the failures related to the difficulty, and the share varies widely between models, from 40% for Claude Haiku 5.5 at xhigh to 86% for Gemini 4 Argon.

The pattern differs between models. GPT-6 Astra and GPT-6.1 Sol most often recognise the problem and then correct for it the wrong way. Grok 4.7, DeepSeek V4.1 Flash and most of the Claude models most often notice the problem in their own output and dismiss it as noise, with Claude Opus 5.5 split evenly between the two. GPT-6 Luna most often never checks at all, and along with Claude Haiku 5.5 at xhigh it is the model least likely to have the evidence in front of it. Gemini 4 Argon fails like the GPT-6 models, most often by handling the problem the wrong way, while GLM 5.3 and Gemini 3.8 Flash most often dismiss it. Claude Haiku 5.5 at xhigh, which stops early, most often never checks, and Kimi K3 is split between never checking and dismissing what it sees.

The five tasks no model solved

Five tasks were never solved by any model. The chart below shows how often models passed tasks with each kind of difficulty, and how many tasks of each kind went unsolved.

Three of the five unsolved tasks have the same kind of difficulty, surprises outside the data (D3), which is also the hardest kind overall. In each of them, an effect that barely shows in the data the agent is given takes over in the conditions where the answer is needed. The other two involve a biased sample and a look-alike. One of the five is an oceanography task in which the agent estimates how much the ocean warmed from the readings of drifting floats. Over the years the floats drifted into warmer water, so a simple trend shows strong warming that mostly reflects where the floats went, and a researcher who plotted where the floats were each year would have seen the drift. Each of these is a known pitfall in its field, and each task has a reference solution that lands inside the correct range, so all five can be solved. The models simply never checked whether the standard analysis applied.

Warning the model does not solve the problem

If models fail because they do not think to check, telling them what to watch for should help. To test this, we reran GPT-6.1 Sol on all 50 tasks with one extra sentence after the instruction. The sentence names the general kind of difficulty without saying where it is in the task, and every task with the same kind of difficulty gets the same sentence.

We compared the new trials with Sol's trials on the leaderboard, three per task in each, and audited them the same way as the leaderboard, without the auditors knowing which trials had the warning.

With the warning, Sol's pass rate rose from 29% to 39%, the net of gains on 18 tasks and losses on 10 others. Sol does not get the same result every time it attempts a task, and with only three trials per task, one lucky or unlucky trial moves a task's pass rate by a third. Some of these ups and downs are therefore chance, so the 95% interval on the overall change runs from 0 to +20 points, and the data are also consistent with no gain at all.

Most of the gain came from three tasks that Sol never solved without the sentence and always solved with it, two with biased samples and one with a quirky instrument. On the other 47 tasks its pass rate rose by four points, from 31% to 35%. On the two biased-sample tasks, the warning led Sol to model what the data were missing, and it passed every trial.

Biased samples show the largest gain partly because there are only five such tasks, so each task that goes from no passes to three moves the group's pass rate by 20 points. Those two tasks alone account for 40 of its 53 points.

The warning made Sol look for the problem, but did not help it once it had looked. Failures where Sol never saw the problem (F1 or F4) fell from 31 to 20, which is still a quarter of its failures from the difficulty even though it had been told what kind of problem to look for. In 11 of those 20 it did check, but on a part of the data where the problem cannot show (F4). Failures after Sol had seen the problem stayed at 61, though more of them became dismissals (F2, from 14 to 21) and fewer were corrections made the wrong way (F3, from 47 to 40). A warning cannot fix a correction made with the wrong model. Sol's main gap on these tasks is not noticing the problem, which a prompt can partly fix, but knowing what to do once it has noticed.

Quality guarantees

A hard benchmark is easy to build badly. A task can be hard because the instruction is ambiguous, because the grader is wrong, or because the answer needs knowledge nobody would have. We treat all three as defects, and we audit for them before a task is built, before it is released, and on every trial that counts toward the board.

Every task declares who should be able to solve it. Whether a task is underspecified depends on who is reading it, so each spec names the researchers it is written for and what they are expected to know. Anything the task needs beyond that must be stated in the instruction, since otherwise the task would be unfair.

Reviews before release. Separate reviewers check each task for leaks, scientific validity, specification and fairness. The known wrong approaches are run and must land outside the range, and correct routes must land inside it.

Re-audit after release. Before assembling the benchmark, we re-audited every released task to make sure it is correct: that correct methods pass, that wrong ones fail, that the instruction is unambiguous and that the grader works.

Every trial is audited. Passes and failures alike are read in full, and when an audit finds a real leak, we fix the task and rerun every model on it.

How it was built

Every task goes through one automated pipeline with human decisions at the points where judgment is needed.

Human seeds

Every task starts from a real problem, taken from one of the three sources in the diagram above. We did not reuse any task from the original benchmark, and every task is a new problem with its own data and grader.

Synthetic generation

A panel of models proposes task ideas from each seed. We screen them for novelty and for difficulty that comes from the science rather than from unrelated hurdles, such as file formats or tooling. The surviving ideas become specs: a contract that states the setting, the researchers the task is written for, the postulated difficulty, the grade, the valid routes and the wrong approaches that must fail.

A builder agent then writes the task: a data generator with hidden truth, the grader, the reference solution and solutions for the known wrong approaches. Fixed quality gates then check that the reference solution lands inside the correct range and that each wrong approach lands outside it.

Adversarial hardening

A frontier model then attempts the built task several times. A diagnoser reads each trial and decides whether it failed because of the postulated difficulty, solved it, or failed for something unrelated. A task is kept only if the model fails because of the postulated difficulty and scores clearly worse than the informed solvers. If the model solves it, the task goes back with what the model did, and the spec is rewritten and the task rebuilt to make the difficulty harder without making it unfair. Every rebuilt version goes through the same reviews again, including the checks for fairness and for ways to game the grader.

Auditing

An agentic auditor tries to break each candidate: it looks for leaks, ambiguity, a grade that rewards the wrong thing, and knowledge that a researcher working in the domain would not have. Released tasks are re-audited, and every trial on the leaderboard gets a trajectory audit, with an extra leak audit for every pass. Any task that fails an audit is fixed before it goes back on the board.

What's next

Our results suggest that today's models cannot yet be trusted to do scientific analysis on their own. Even the best model failed more than half of the trials. How the models failed also shows why they cannot yet work alone. In about two thirds of the failed trials, the model had already seen the problem, in a number it printed or a check it ran, and then either dismissed it as noise or handled it the wrong way. Its code still ran and its results looked reasonable, so a researcher relying on it would have had no warning that the answer was wrong.

This is the gap that Vetto Science Bench measures. Frontier models can write the code that research needs, but they do not yet have the judgment to notice when the standard method does not fit the data in front of them. Because every task has a known answer and a difficulty that a careful researcher would catch, the benchmark shows how far models are from that standard, and it will show when they get there.

We will add new models to the leaderboard as they are released, and we will keep auditing the tasks after release.

Appendix: detailed results

We ran 13 frontier models on 50 tasks, chosen to give a good mix of failure modes, disciplines and difficulty, with three trials per task, using mini-swe-agent. Claude Haiku 5.5 ran at two reasoning settings, so there are 14 configurations in all. By default the agent works in text mode, where the model writes one shell command per step as plain text. Seven models ran in tool mode instead, where each step is a native tool call: DeepSeek V4.1 Flash, GPT-6 Luna and Grok 4.7, which had trouble keeping to the text format, and Gemini 4 Argon, Gemini 3.8 Flash, GLM 5.3 and Kimi K3.

The audit of the five configurations we added last struck eleven passes on six tasks, where the model had reconstructed how the task's data were generated instead of analysing them, for example by fitting the round-number periods of a simulated trajectory. Nine of those passes were Gemini 4 Argon's.

As the chart shows, a model's price is a poor guide to how well it does on these tasks (note that cost is on a log scale). The frontier also has a sharp step. The cheapest models, up to DeepSeek V4.1 Flash at about 60 cents per trial, pass at most one trial in ten, while GPT-6.1 Sol, at about 80 cents, passes more than a quarter. GPT-6 Astra has the highest pass rate, and Gemini 4 Argon comes close at more than three times the cost per trial. GPT-6.1 Sol reaches three fifths of Astra's pass rate at a fifth of its cost, while Claude Opus 5.5 costs almost as much as Astra and passes about half as many trials. At xhigh, Claude Haiku 5.5 is by far the cheapest model on the chart, but it stops after about ten steps and passes a third as often as it does at max. Since prices and reasoning settings differ between providers, small gaps on this chart should not be over-read.

How long models work

For tasks that would take a researcher hours or days, the models stop remarkably early. In text mode, where each step is one shell command, the models need a median of 10 to 19 steps and 4 to 18 minutes of wall-clock time. Claude Fable 5.1 stops remarkably early for a frontier model, usually submitting after about 11 steps and four minutes. Claude Haiku 5.5 is the exception on time when running at max effort, with a median of over an hour, but the extra time comes from long reasoning rather than from more steps: its median trial produces about 725,000 output tokens, roughly ten times as many as Opus or Sonnet. At xhigh, the same model submits after a median of ten steps and six minutes.

The models we ran in tool mode take many more steps, partly because each tool call does less than a full shell command. For most of them, the extra steps do not translate into better results. DeepSeek V4.1 Flash, for example, has a median of 116 steps per trial and still passes fewer than one trial in ten, as do five of the other six models in tool mode. Gemini 4 Argon is the exception. It works longer than any other model, with a median of 104 steps and 72 minutes per trial, and it passes 47% of its trials.

Beyond the five unsolved tasks, the grid shows how thinly the passes are spread. Most tasks were solved by only a handful of models, and none by more than 12 of the 14 configurations. The models also differ in which tasks they get right. GPT-6 Astra and Gemini 4 Argon each failed 12 tasks that at least one other model solved, and seven tasks were solved by only a single model. A higher place on the leaderboard therefore does not mean a model is better at every kind of problem.