Computer Anthology

Introducing Computer Anthology

Agent benchmarks are failing in two ways. The visible one is saturation: on the suites that defined the last two years, frontier pairs cluster within a few points of the ceiling and of each other, and much of what still separates them is noise from tasks that fail correct solutions or pass broken ones. A leaderboard in that state has stopped telling anyone which model to reach for.

The quieter failure is that benchmarks get built as one-shot projects. Months go into authoring, review, and calibration, a static artifact ships, the frontier catches up, and the artifact is spent with little of the machinery surviving to build the next one. Building a single harder set buys less time than it used to, since recent generations have consumed most of the added difficulty within months.

Computer Anthology attacks the second failure, on the theory that it makes the first one manageable. It is a family of benchmarks, each measuring one computer skill (terminal work, GUI computer use, reverse engineering, program synthesis, repository-scale engineering) rather than collapsing everything into a single agent score. What we are building to produce them is closer to a data engine than a dataset: an ecosystem of agents, proprietary models, humans in the loop, and infrastructure that is reused across benchmarks and keeps improving as models improve. Each benchmark assembles that ecosystem into its own pipeline, since different skills need different environments, verifiers, and calibration, so every entry starts from stronger machinery than the last and a saturated set means assembling its successor rather than starting the work over.

Every entry keeps tasks held out, never trained on and never released while they are still measuring something, so a set can be re-scored across model generations without contamination. Some entries may be published in full, and held-out tasks can open up later, as a benchmark saturates or a new version supersedes it. Whatever we release ships in an open format when possible, either Harbor or cua.

Terminal Tasks v1.0

The first entry is 100 self-contained, verifier-graded tasks an agent completes from a terminal, rebuilding the Terminal-Bench formula: on Terminal-Bench 2.0 frontier pairs now cluster near 85% and the median task is passed in roughly seven attempts out of ten. Four properties define v1.0, each with an operational definition in Methodology:

• Hard. Every task was calibrated against a frontier model while it was being written, and accepted only when that model passed at most 60% of its calibration trials. The best configuration we have measured passes 62.0% of first attempts, and 17 tasks are solved on at most one attempt in ten.
• Fair. A hard task is trivially easy to write by making it broken. Every task in the set carries empirical evidence that it is solvable from its own materials, and every trajectory is read by a judge working from a defect taxonomy, with a human ruling on anything it flags.
• Deterministically verified. Each task ships a sandboxed verifier that writes a binary reward. No LLM judge grades any attempt, and results reproduce run over run.
• Human-reviewed. Vetto researchers read the instruction, environment, verifier, and reference solution of all 100 tasks before they entered the set.

Each task is a job a competent practitioner could plausibly be handed. The set spans 17 categories and 11 languages and runtimes, weighted toward data science, ML, optimization, and model-training work, and it grades whether the artifact works within the stated constraints, not code quality or style.

v1.0 is already dated, in the way every benchmark is: it was calibrated in the era of Claude Opus 4.6 and GPT-5.4, and a couple of model generations took it from roughly 15% pooled to 62.0%. We read that as a good sign. A set built from the failures of two named models could have turned out to be a pile of quirks, and instead each new generation has climbed on it steadily, which suggests the tasks track something the labs are deliberately working on. We use a few techniques to try to keep those biases out. The yardstick is working, and it is also counting down.

The tasks stay held out while they remain useful for measurement, so we publish the structure, the protocol, and the results. We also run models on request, including a lab's private model under its own scaffold (tasks never leave our infrastructure).

Results

Every number below comes from one protocol: 5 independent trials per task per configuration, 500 trials per configuration, run on Harbor and graded by each task's own verifier. Each trial gets a freshly built, isolated container with the task's declared CPU and memory. Agents get one hour of wall clock, raised from shorter budgets until the limit stopped shaping most outcomes. We evaluate and report model and agent as a pair. Trials that hit the wall are graded normally rather than being dropped, and infrastructure errors are retried. This holds every denominator at 500. Intervals are trial-level bootstrap 95% CIs, and same-model scaffold comparisons are paired on per-task differences.

Reasoning effort is part of each configuration, but we did not force every model to use either a common tier or its highest available tier. Providers define reasoning levels differently, so labels such as medium and high are not directly comparable across model families. Instead, we evaluated the experience a user would get by default from each agent-model combination. Agent defaults take precedence, with the model provider's default being used on Terminus-2 as it doesn't force a reasoning level. The parenthetical labels below show those effective settings. When no parenthetical appears, the run used an unspecified model provider default.

The leaderboard

The 28 configurations span 2.8% to 62.0% and average 30.6%. Difficulty is spread rather than bimodal: the median task is passed on 29.6% of the 140 attempts it has received, 17 tasks sit at or below 10%, and another 19 at or above 50%. Three patterns stand out.

Spend does not explain capability. MuseSpark 1.1 produces the most output tokens per trial on the board, 174k, and passes 39.0%; GPT-5.6 Sol under Codex CLI passes 58.0% on 12.9k, and DeepSeek V4 Pro spends 72k to pass 10.4%. Step counts inside the Terminus-2 rows, where step granularity is identical across models, agree: Claude Fable 5 scores 50.8% on a median of 15 steps per trial, Gemini 3.5 Flash 22.8% on 41, Devstral 2 2.8% on 39. Failing trials are not systematically longer than passing ones either, with medians differing by a few steps in most configurations and the widest gap under the MuseSpark configuration (19 steps when passing, 36 when failing).

Use the buttons to switch the x-axis between output tokens, cost per trial, and median steps.

The full agent configuration can be worth double-digit points. Moving from Terminus-2 to the native scaffold raises GPT-5.5 by 13.0 points pass@1 and GPT-5.6 Sol by 6.4, but those comparisons also move from medium to high reasoning. The cleaner same-effort Claude comparisons point both ways: Opus 4.8 gains 8.2 points under Claude Code, while the board's strongest configuration is Opus 5 under Terminus-2, 8.0 points above the same model under Claude Code. All four differences are separated under a paired bootstrap on per-task differences. The deltas for Claude Fable 5, Gemini 3.5 Flash, and Gemini 3.1 Pro are within noise.

The harness effect on Opus 5 is not uniform. Under Claude Code, the model more often finds the core fix and then keeps going, refactoring adjacent interfaces, tightening constraints, or revising validation machinery that the task expects it to preserve. That extra work can turn a correct repair into a contract violation, but it pays off when broader investigation is useful: on Data Processing, Claude Code scored 35% to Terminus-2's 20%. Terminus-2 more often keeps the work narrow, but can spend too long in open-ended exploration; the 8-point gap is the net of those opposing tendencies, not evidence that one scaffold is uniformly better.

A couple of model generations moved the pooled rate from about 15% to above 60%. Holding the agent fixed at Terminus-2 and swapping only the model, the Claude line runs from Opus 4.6 at 14.8%, through Opus 4.8 at 29.8% and Fable 5 at 50.8%, to Opus 5 at 62.0%. Among the GPT configurations that both use medium reasoning, GPT-5.5 scores 38.6% and GPT-5.6 Sol 51.6%. This is the measurement the benchmark exists for: a fixed yardstick for how much verifiable computer-control capability each generation adds.

Results by category

Use the filter to add or remove configurations.

We do not report per-category results for categories holding five or fewer tasks, since those cells are not statistically meaningful. Those trials still count toward every board-wide number.

Debugging is the hardest category on board average at 19%, and Optimization, Software Engineering, and Data Querying the easiest, at 34% to 36%. Models do not agree on that ordering, though: the median rank correlation between any two configurations' category profiles is 0.20, so what a model finds hard is mostly specific to the model. The same goes for the harness. Claude Opus 5 doubles on Debugging when moved from Claude Code to Terminus-2 (26% to 51%) and drops from 35% to 20% on Data Processing; GPT-5.6 Sol collapses from 77% to 27% on Software Engineering under the same move, while gaining on Model Training (44% to 58%) and Data Science (40% to 51%).

One caveat on reading any single cell: scoring is all-or-nothing per trial, so a cell is the share of trials that passed the task's full verifier, and many failing trials clear most of the underlying checks. On the 17 tasks solved on at most one attempt in ten, a failing trial still clears a median of 70% of the verifier's individual checks. We track partial rewards internally and report the binary outcome.

Provider safety filters

Some legitimate defensive-security and forensics tasks trigger providers' cyber-safety filters. We classify a trial as filtered only when its artifacts contain an explicit refusal or safety signal. A terminal filter receives no reward and counts as a failure, while Claude Code can continue through a fallback model.

We mirrored Claude Code's fallback for the Anthropic trials under Terminus-2: when Fable 5 or Opus 5 refuses, we route the continuation to Opus 4.8. There is no fallback from Opus 4.8, so when it refuses, the trial terminates; we added no fallback for non-Anthropic models because their native agents provide no fallback behavior to mirror. This makes the two Anthropic harness paths more comparable, but a continued trial no longer measures the primary model alone.

Each bar is the percentage of that configuration's 500 trials with a confirmed safety-filter event, split by whether execution continued. The remaining 19 configurations recorded none.

Across the board, 191 of 14,000 trials (1.4%) were explicitly filtered: 165 (86.4% of filtered trials) continued on fallback and 26 (13.6%) terminated. Fable 5 triggered a filter on 15.6% of Claude Code trials and 14.6% of Terminus-2 trials, all of which continued under Opus 4.8 and therefore partly measure the fallback model. Those trials passed 38.5% and 42.5% of the time, against 56.9% and 52.2% for the corresponding unfiltered trials.

Anatomy of the hardest tasks

Seven tasks were solved on no more than 7 of their 140 attempts. We read the trajectories on all of them, and the failures are strikingly uniform. Each of those tasks turns on a single judgment about what the environment means rather than on breadth or obscurity: whether a cache holds raw or normalized spectra, whether two deliveries carrying identical payloads are one event or two, which of two completed retries is the canonical one, whether durable facts belong inside the checkpoint or beside it. What sinks these trials is rarely an inability to work the problem out; it is an assumption adopted early, never checked, and carried all the way to a confident sign-off.

Partial credit shows how close those trials run to correct. On the state-machine generator a third of all trials finish one check short of a full pass, and the identity-stitching task averages 25 of its 30 checks. What separates the few winners is not more searching but one extra round of doubt: each of them built a test for the thing everyone else assumed. Two runs of the same model, under the same harness and the same prompt, diverged only because one went back to the seed data and reversed its own deduplication rule. Across every trajectory we read we found no attempt to game a verifier, edit a test, or hardcode an expected value.

We read these as hard rather than broken, and that is a distinction we hold ourselves to at this end of the set, because a task nothing solves is a claim about us until we can show otherwise. The final review of the set went looking for tasks that fail that test, reading every trial of every candidate at the bottom of the range. Some turned out to be genuinely defective, requiring behavior their instructions never specified, rejecting valid solutions, or rewarding shortcuts rather than capability, so we replaced them rather than presenting them as difficulty. Nothing on the board is unsolved today: every task has been passed by at least one configuration, and the hardest one standing is passed on 2 of its 140 attempts.

One task, end to end

Here is one task in full: go-netscan-pattern-budget, an Optimization task that 15 of 25 showcase trials solved (49 of its 140 trials in all). It is not in the benchmark. Publishing a held-out task burns it, so this one was retired from the scored set and replaced from the same pipeline, as were the two other sample tasks in the explorer below.

The agent receives this instruction, verbatim, and a shell in a small Go repository:

The netscan-pattern generator emits patterns that are correct but too long: on the production configs the move sequence overruns the firmware's 64-byte pattern slot, and netscan-pattern exits with a budget error. It needs to emit the shortest sequence that still covers every cell of the rows × cols address space and returns the probe to (0, 0) after a whole number of repetitions, so it fits the slot on every (rows, cols) the fleet uses. The netscan-pattern binary reads -rows and -cols and prints the pattern. make test runs a small visible suite as a smoke check on the typical small cases; production sign-off needs the generator correct across the full configuration set, not only the ones the visible suite covers. The default build is make.

The probe sweeps a rows × cols address space that wraps on both axes, and the shipped generator walks it the obvious way, one column at a time, emitting a sequence as long as the row count. Behind that framing is a real result the agent has to derive: the shortest pattern that covers a torus and returns to the origin has length max(2, gcd(rows, cols)), because each repetition displaces the probe by a fixed vector and one move is needed per coset of the subgroup that vector generates. Getting the bound is the easy half. Most configurations reach it with a monotone staircase, D^a R^b, but the graded set contains eleven where no such split exists at all: every split coprime to the row count shares a factor with the column count. Those require a word with back-steps in it, chosen so its prefixes land in distinct cosets. The workspace ships a brute-force coverage checker the agent can test candidates against, and two plausible-looking helper modules computing min and lcm that are both the wrong quantity. The visible make test passes on the unmodified broken generator.

There is nothing to submit. The verifier rebuilds the binary from the agent's source, runs the agent's own visible suite, and then invokes the binary on 25 held-out configurations, checking for each one that the emitted pattern covers the space, returns to the origin, fits the 64-byte slot, and has length exactly max(2, gcd). Coverage is checked algebraically rather than by simulation, which is how grids of 213 million cells are graded in seconds. The configuration set is built to close off memorization: it includes axis swaps and scalar multiples of each obstructed grid, so a generator that hardcodes one winning word per gcd value fails on the transposes, and one obstruction family never appears in the visible material at all.

Five configurations attempted it 5 times each: Claude Opus 5 and Claude Fable 5 under Claude Code, GPT-5.6 Sol and GPT-5.5 under Codex CLI, and Gemini 3.5 Flash under Terminus-2.