Computer Anthology

Introducing Computer Anthology

Agent benchmarks are failing in two ways. The visible one is saturation: on the suites that defined the last two years, frontier pairs cluster within a few points of the ceiling and of each other, and much of what still separates them is noise from tasks that fail correct solutions or pass broken ones. A leaderboard in that state has stopped telling anyone which model to reach for.

The quieter failure is that benchmarks get built as one-shot projects. Months go into authoring, review, and calibration, a static artifact ships, the frontier catches up, and the artifact is spent with little of the machinery surviving to build the next one. Building a single harder set buys less time than it used to, since recent generations have consumed most of the added difficulty within months.

Computer Anthology attacks the second failure, on the theory that it makes the first one manageable. It is a family of benchmarks, each measuring one computer skill (terminal work, GUI computer use, reverse engineering, program synthesis, repository-scale engineering) rather than collapsing everything into a single agent score. What we are building to produce them is closer to a data engine than a dataset: an ecosystem of agents, proprietary models, humans in the loop, and infrastructure that is reused across benchmarks and keeps improving as models improve. Each benchmark assembles that ecosystem into its own pipeline, since different skills need different environments, verifiers, and calibration, so every entry starts from stronger machinery than the last and a saturated set means assembling its successor rather than starting the work over.

Every entry keeps tasks held out, never trained on and never released while they are still measuring something, so a set can be re-scored across model generations without contamination. Some entries may be published in full, and held-out tasks can open up later, as a benchmark saturates or a new version supersedes it. Whatever we release ships in an open format when possible, either Harbor or cua.

Terminal Tasks v1.0

The first entry is 100 self-contained, verifier-graded tasks an agent completes from a terminal, rebuilding the Terminal-Bench formula: on Terminal-Bench 2.0 frontier pairs now cluster near 85% and the median task is passed in roughly seven attempts out of ten. Four properties define v1.0, each with an operational definition in Methodology:

• Hard. Every task was calibrated against a frontier model while it was being written, and accepted only when that model passed at most 60% of its calibration trials. The best configuration we have measured passes 61.8% of first attempts, and 17 tasks are solved on at most one attempt in ten.
• Fair. A hard task is trivially easy to write by making it broken. Every task in the set carries empirical evidence that it is solvable from its own materials, and every trajectory is read by a judge working from a defect taxonomy, with a human ruling on anything it flags.
• Deterministically verified. Each task ships a sandboxed verifier that writes a binary reward. No LLM judge grades any attempt, and results reproduce run over run.
• Human-reviewed. Vetto researchers read the instruction, environment, verifier, and reference solution of all 100 tasks before they entered the set.

Each task is a job a competent practitioner could plausibly be handed. The set spans 17 categories and 11 languages and runtimes, weighted toward data science, ML, optimization, and model-training work, and it grades whether the artifact works within the stated constraints, not code quality or style.

v1.0 is already dated, in the way every benchmark is: it was calibrated in the era of Claude Opus 4.6 and GPT-5.4, and a couple of model generations took it from roughly 15% pooled to 61.8%. We read that as a good sign. A set built from the failures of two named models could have turned out to be a pile of quirks, and instead each new generation has climbed on it steadily, which suggests the tasks track something the labs are deliberately working on. We use a few techniques to try to keep those biases out. The yardstick is working, and it is also counting down.

The tasks stay held out while they remain useful for measurement, so we publish the structure, the protocol, and the results. We also run models on request, including a lab's private model under its own scaffold (tasks never leave our infrastructure).

Results

Every number below comes from one protocol: 5 independent trials per task per configuration, 500 trials per configuration, run on Harbor and graded by each task's own verifier. Each trial gets a freshly built, isolated container with the task's declared CPU and memory. Agents get one hour of wall clock, raised from shorter budgets until the limit stopped shaping most outcomes. We evaluate and report model and agent as a pair. Trials that hit the wall or die to infrastructure count as failures rather than being dropped, which holds every denominator at 500. Intervals are trial-level bootstrap 95% CIs, and same-model scaffold comparisons are paired on per-task differences.

The leaderboard

The 28 configurations span 2.8% to 61.8% and average 30.5%. Difficulty is spread rather than bimodal: the median task is passed on 29.6% of the 140 attempts it has received, 17 tasks sit at or below 10%, and another 19 at or above 50%. Three patterns stand out.

Spend does not explain capability. MuseSpark 1.1 produces the most output tokens per trial on the board, 174k, and passes 39.0%; GPT-5.6 Sol under Codex CLI passes 58.0% on 12.9k, and DeepSeek V4 Pro spends 72k to pass 10.4%. Step counts inside the Terminus-2 rows, where step granularity is identical across models, agree: Claude Fable 5 scores 50.8% on a median of 15 steps per trial, Gemini 3.5 Flash 22.8% on 41, Devstral 2 2.8% on 38. Failing trials are not systematically longer than passing ones either, with medians differing by a few steps in most configurations and the widest gap under MuseSpark 1.1 + Terminus-2 (19 steps when passing, 36 when failing).

Use the buttons to switch the x-axis between output tokens, cost per trial, and median steps.

The harness can be worth double-digit points. Moving from Terminus-2 to its native scaffold, GPT-5.4 gains 26.0 points pass@1, GPT-5.4 Mini 17.8, GPT-5.5 13.0, Claude Opus 4.8 8.2, and GPT-5.6 Sol 6.4, every one of them separated under a paired bootstrap on per-task differences. The native scaffold does not always win, though: the board's strongest configuration is Claude Opus 5 under Terminus-2, 7.8 points above the same model under Claude Code, and it is the first model we have measured that does significantly better outside its own scaffold. The Claude Fable 5 and the two Gemini deltas are within noise.

A couple of model generations moved the pooled rate from about 15% to above 60%. Holding the agent fixed at Terminus-2 and swapping only the model, the Claude line runs 14.8%, 29.8%, 50.8%, 61.8% (Opus 4.6, Opus 4.8, Fable 5, Opus 5) and the GPT line 12.0%, 38.6%, 51.6% (GPT-5.4, GPT-5.5, GPT-5.6 Sol). This is the measurement the benchmark exists for: a fixed yardstick for how much verifiable computer-control capability each generation adds.

Results by category

Use the filter to add or remove configurations.

We do not report per-category results for categories holding five or fewer tasks, since those cells are not statistically meaningful. Those trials still count toward every board-wide number.

Debugging is the hardest category on board average at 19%, and Optimization, Software Engineering, and Data Querying the easiest, at 34% to 36%. Models do not agree on that ordering, though: the median rank correlation between any two configurations' category profiles is 0.20, so what a model finds hard is mostly specific to the model. The same goes for the harness. Claude Opus 5 doubles on Debugging when moved from Claude Code to Terminus-2 (26% to 51%) and drops from 35% to 20% on Data Processing; GPT-5.6 Sol collapses from 77% to 27% on Software Engineering under the same move, while gaining on Model Training (44% to 58%) and Data Science (40% to 51%).

One caveat on reading any single cell: scoring is all-or-nothing per trial, so a cell is the share of trials that passed the task's full verifier, and many failing trials clear most of the underlying checks. On the 17 tasks solved on at most one attempt in ten, a failing trial still clears a median of 70% of the verifier's individual checks. We track partial rewards internally and report the binary outcome.

Anatomy of the hardest tasks

Seven tasks were solved on no more than 7 of their 140 attempts. We read the trajectories on all of them, and the failures are strikingly uniform. Each of those tasks turns on a single judgment about what the environment means rather than on breadth or obscurity: whether a cache holds raw or normalized spectra, whether two deliveries carrying identical payloads are one event or two, which of two completed retries is the canonical one, whether durable facts belong inside the checkpoint or beside it. What sinks these trials is rarely an inability to work the problem out; it is an assumption adopted early, never checked, and carried all the way to a confident sign-off.

Partial credit shows how close those trials run to correct. On the state-machine generator a third of all trials finish one check short of a full pass, and the identity-stitching task averages 25 of its 30 checks. What separates the few winners is not more searching but one extra round of doubt: each of them built a test for the thing everyone else assumed. Two runs of the same model, under the same harness and the same prompt, diverged only because one went back to the seed data and reversed its own deduplication rule. Across every trajectory we read we found no attempt to game a verifier, edit a test, or hardcode an expected value.

We read these as hard rather than broken, and that is a distinction we hold ourselves to at this end of the set, because a task nothing solves is a claim about us until we can show otherwise. The final review of the set went looking for tasks that fail that test, reading every trial of every candidate at the bottom of the range. Some turned out to be genuinely defective, shipping without a fixture their own tests require or enforcing a convention their instructions never state, and we replaced them rather than present them as difficulty. Nothing on the board is unsolved today: every task has been passed by at least one configuration, and the hardest one standing is passed on 2 of its 140 attempts.

One task, end to end

Here is one task in full: go-netscan-pattern-budget, an Optimization task that 15 of 25 showcase trials solved (48 of its 140 trials in all). It is not in the benchmark. Publishing a held-out task burns it, so this one was retired from the scored set and replaced from the same pipeline, as were the two other sample tasks in the explorer below.

The agent receives this instruction, verbatim, and a shell in a small Go repository:

The netscan-pattern generator emits patterns that are correct but too long: on the production configs the move sequence overruns the firmware's 64-byte pattern slot, and netscan-pattern exits with a budget error. It needs to emit the shortest sequence that still covers every cell of the rows × cols address space and returns the probe to (0, 0) after a whole number of repetitions, so it fits the slot on every (rows, cols) the fleet uses. The netscan-pattern binary reads -rows and -cols and prints the pattern. make test runs a small visible suite as a smoke check on the typical small cases; production sign-off needs the generator correct across the full configuration set, not only the ones the visible suite covers. The default build is make.

The probe sweeps a rows × cols address space that wraps on both axes, and the shipped generator walks it the obvious way, one column at a time, emitting a sequence as long as the row count. Behind that framing is a real result the agent has to derive: the shortest pattern that covers a torus and returns to the origin has length max(2, gcd(rows, cols)), because each repetition displaces the probe by a fixed vector and one move is needed per coset of the subgroup that vector generates. Getting the bound is the easy half. Most configurations reach it with a monotone staircase, D^a R^b, but the graded set contains eleven where no such split exists at all: every split coprime to the row count shares a factor with the column count. Those require a word with back-steps in it, chosen so its prefixes land in distinct cosets. The workspace ships a brute-force coverage checker the agent can test candidates against, and two plausible-looking helper modules computing min and lcm that are both the wrong quantity. The visible make test passes on the unmodified broken generator.

There is nothing to submit. The verifier rebuilds the binary from the agent's source, runs the agent's own visible suite, and then invokes the binary on 25 held-out configurations, checking for each one that the emitted pattern covers the space, returns to the origin, fits the 64-byte slot, and has length exactly max(2, gcd). Coverage is checked algebraically rather than by simulation, which is how grids of 213 million cells are graded in seconds. The configuration set is built to close off memorization: it includes axis swaps and scalar multiples of each obstructed grid, so a generator that hardcodes one winning word per gcd value fails on the transposes, and one obstruction family never appears in the visible material at all.

Five configurations attempted it 5 times each: Claude Opus 5 and Claude Fable 5 under Claude Code, GPT-5.6 Sol and GPT-5.5 under Codex CLI, and Gemini 3.5 Flash under Terminus-2.

Hover any dot to read the agent's step at that moment.

Claude Opus 5 passed 4 of 5, GPT-5.6 Sol and GPT-5.5 4 of 5 each, Claude Fable 5 3 of 5, and Gemini 3.5 Flash 0 of 5. Every trial built cleanly and left the visible suite green; nobody broke the repository they were handed. Three of the ten failures are not reasoning failures either: they are Claude Code trials the provider cut off mid-run, twelve to fourteen minutes in, with the workspace still untouched and the reasoning demonstrably on track. We count them as failures rather than dropping them, which is why this task's Claude Code numbers understate both models.

The rest of the board fails in one shape. Across all 140 trials, 21 of them, spanning twelve different model-and-harness configurations, emitted the byte-identical wrong answer DRRRRRRRRRRRRRRRR. That is not a coincidence: it is the true optimum of the wrong search space. Seventeen moves is the shortest staircase for that grid, and every agent that assumed the alphabet had two letters instead of four converged on it. Another 20 trials never got past a gentler version of the same trap, a grid where a staircase does reach the bound but only if it starts with seven down-moves instead of the obvious one. The single missing insight is small and sharp, and the winners all state it explicitly before writing any code: the length bound still holds, so allow back-steps. Their answers look like DRRRDRRRDRRRDRUR. One Opus 5 trial went further and enumerated, from the one obstructed grid visible in its workspace, both of the obstruction families the hidden verifier is built from, including the one it had never seen.

What makes the task worth showing is that the wrong answer is a good answer. DRRRRRRRRRRRRRRRR genuinely covers all 3.8 million cells, genuinely returns to the origin, and sits 47 bytes inside the firmware budget it was asked to fit. Judged against the business outcome the instruction opens with, it is a fix; it fails only the minimality clause in the next sentence. All seven failures that ran to completion signed off with confidence, and one went further: having convinced itself that 17 was the true minimum, it went back and relaxed a minimality assertion it had earlier added to its own test suite so that its suite would agree. Its final report lists the length as verified. The self-verification loop worked perfectly and was pointed at the wrong hypothesis, which is the argument for grading against a verifier the agent never sees.

Explore the task yourself

Everything above is summarized from the raw artifacts; the explorer beside the trajectory chart holds the artifacts themselves, for this task and two more retired samples spanning Data Querying and File Operations. It shows each task as the agent saw it, the full verifier, and all 25 trajectories step by step: stated reasoning, every command, and what came back.

The verifier sources are published here only because these three tasks are retired; for every live task they stay hidden. The one remaining redaction is each task's build-time seeding script, which the agent's container never contains.

We keep this level of instrumentation for all 100 tasks. Every leaderboard number above is backed by trajectories like these.

What is in the set

Each task gives the agent a working directory, an instruction.md, and a definition of done enforced by a hidden verifier, with every requirement the verifier checks stated in the instruction. The agent works in a sandboxed Linux environment under fixed resource limits (1 CPU and 2 GB RAM by default, plus storage and wall-clock caps) and is graded on the artifact it produces. No task depends on the public internet: even the web tasks run against local sites and localhost APIs, so results do not change when external services do. Containers may have network access for the agent's own tooling, but no verifier checks anything outside the sandbox.

The 100 tasks span 17 categories, allocated in three tiers rather than in proportion to where tasks are easiest to author:

Tier	Categories	Tasks	Rationale
Moat	Data Science, Machine Learning, Optimization, Model Training	44	Research, data, and training work that other agent benchmarks largely lack.
Core	Software Engineering, Data Querying, Data Processing, Security, System Administration	32	Essential computer-control work; SWE de-emphasized because code-specific suites already cover it.
Base	Debugging, Scientific Computing, File Operations, Video Processing, Web Browsing, Personal Assistant, Games, Mathematics	24	Quintessential or niche surfaces, kept at a representative floor.

The set runs against real engines, including PostgreSQL, SQLite, MongoDB, nginx, PAM, git, ffmpeg, and ext4/debugfs, across 11 languages and runtimes. The intent is to measure computer control, not only Python fluency.

Across categories the tasks cluster into a handful of overlapping capability themes, and most exercise several, so the counts sum to more than 100. The largest theme, diagnosing and repairing a subtle defect in working-looking code, covers more than a third of the set.

21 tasks impose production-style constraints (bounded memory, out-of-core processing, latency SLAs, deterministic or idempotent output), and on those a correct but resource-blind solution fails. One live Optimization task hands the agent a working threat-intel scanner that walks a 50,000-entry blocklist once per line of a multi-gigabyte gateway log. The headline idea (a multi-pattern string index) is standard; the difficulty is the contract around it, from hostname-label boundaries to byte-identical output across input modes, verified against a hash under a peak-RSS cap.

How the tasks were built

Hard and fair, operationally

Hard means anchored to a low frontier pass rate, and the anchoring rule was explicit and single: every task was calibrated during authoring against one of two then-current frontier pairs, 62 tasks against GPT-5.4 + Codex and 38 against Claude Opus 4.6 + Terminus-2, and accepted only when its calibration target passed at most 60% of calibration trials. A set this hard for the calibration frontier leaves headroom for the next generation, so the leaderboard keeps discriminating instead of pinning at 100%. Selecting tasks because named models fail them also biases the set toward those models' weaknesses, which we measure in Selection on failure.

Fair is the harder property, and we would rather be transparent about why than claim we have solved it. A fair task clears three bars at once. It must not reject correct work, so every requirement the verifier checks is stated in the instruction and no check pins an incidental choice of the reference solution. It must not accept incorrect work, so there is no path to the reward through hardcoding, guessing, or reading the answer out of the environment. And it must be worth measuring, grounded in work that matters in real use rather than a puzzle designed backward from a verifier. These bars pull against each other: every check added to block a false positive is a new chance to create a false negative, and the realism that makes a task worth measuring multiplies the places ambiguity can hide. There is no oracle for any of the three, so we use instruments that cover one another's blind spots: human expert review in several independent stages, LLM judges armed with a human-built taxonomy of recurring defects, and empirical evidence from two directions, easing (does information derivable from the task's own materials move the pass rate) and semantic perturbation (does the difficulty survive rewording).

The authoring pipeline

The current process is the third iteration. Experts-only authoring cost roughly 20 hours per task and still produced ambiguous specifications and verifier false positives at a high rate; adding agents after the fact caught real defects but made for a slow loop where errors compounded across rounds. What worked was putting the agents inside the loop, packaged into an SDK the authors use directly for debugging environments, generating boilerplate, and running calibration models against drafts. It beat both expert-only authoring with model-based checking and synthetic generation with human review, and it lets experts spend their hours on what agents handle poorly: choosing problems from real production incidents, deciding what makes a task genuinely hard, catching the ambiguity a verifier would miss.

Each step is gated by a different group, and no task reaches the benchmark without clearing all of them:

1. Authoring. Domain experts create each task inside the SDK, running the calibration models against drafts as they iterate. The 60% difficulty bar is measured here, against live calibration runs. Around 5 hours of active human time per task, a quarter of the expert-only cost.
2. Independent review, two reviewers in parallel. Both work independently of each other and of the author, without SDK access but with frontier-model trajectories in hand. A task typically survives up to three rounds of findings before both sign off. Around 2 hours per task.
3. Curation. Vetto-side triage: we audit fairness, difficulty, and quality, with AI assistance on the analysis, and select which tasks enter the candidate pool.
4. Hinting. Every candidate gets optional hints and an easing run measuring how much they lift the pass rate, the solvability evidence described below. The benchmark ships hints-off.
5. Judge scoring. The LLM judge reads everything produced so far (instruction, environment, reference solution, verifier, calibration trajectories, pass rates, hint lift) and writes structured metadata: a task-quality score, a difficulty estimate, capability tags.
6. Human review. A Vetto researcher makes the final ready-or-not call on the task's full history plus the judge's metadata, and after any fix the judge re-scores, which is the loop at the bottom of the diagram. Every task in the held-out set passed this stage manually. The errors researchers catch here feed back into the defect taxonomy: we add categories and few-shot examples and rework the judge, so each failure mode found by hand becomes one the judge screens for afterward.

Two instruments for fairness at scale

Human review does not reach every trajectory of every candidate, so two instruments carry the load where it does not.

The first is judges with a defect taxonomy. Unaided LLM judges are unreliable reviewers: asked open-endedly whether a task is fair, they miss the defects that matter and flag things that are not defects. Structure is what made them useful. Across audit sweeps of hundreds of submissions we catalogued how tasks actually fail: verifiers that grep the source for expected strings instead of testing behavior, absolute timing gates that tie the reward to host hardware, byte-exact output pins on formatting the instruction never specifies, probabilistic concurrency assertions that fail a correct solution some fraction of the time, requirements enforced by the verifier but absent from the instruction. Each entry carries few-shot examples, including near-misses that resemble the defect but are legitimate. Judges prompted with this taxonomy screen every candidate and its calibration trajectories, and everything they flag routes to a human for the ruling. Nothing is accepted or rejected by a judge alone, and the shipped benchmark uses no LLM judge anywhere in scoring.

The second is easing, which exists because a task no model passes is ambiguous evidence: a 0% pass rate cannot distinguish a genuinely hard task from an underspecified, mis-verified, or impossible one, and a benchmark full of unfalsifiable zeros measures nothing. So we author optional hints for every candidate and measure how much they move the pass rate, on the logic that if information legitimately derivable from the task's own materials lifts the rate, a path from prompt to solution exists. A task sitting at 0% is admitted only after easing has demonstrated a legitimate pass. This is evidence, not proof, since a hint-assisted pass does not show the unhinted prompt is free of every ambiguity, but it is a stronger check than the field's default of author intuition and spot checks. The benchmark ships hints-off; the hints exist for calibration, and when tasks like these are used for RL they also give a less capable model a denser reward signal than the unhinted prompt, which would return little but zeros.

Selection on failure: what it costs us

The difficulty bar is adversarial by construction. A task enters the set because a named model failed it, so filtering on the failures of two models over-samples whatever those two happen to be bad at, and difficulty here is not an intrinsic property of the tasks: it was manufactured against GPT-5.4 + Codex and Claude Opus 4.6 + Terminus-2. Two distortions follow. The set may over-represent those two models' failure modes, so other families get scored on terrain chosen to be hard for someone else, in either direction. And a task can clear the 60% bar not because the underlying work is hard but because it trips an idiosyncrasy of the target: a formatting habit, a tool-use blind spot, a family-specific prior. Such a task measures a quirk, not capability.

To size this, we split the 100 tasks by calibration target and compared today's pass rates on each subset.

The GPT-calibrated tasks are simply harder for everyone: all but one of the 28 configurations score higher on the Opus subset, a board-average gap of 14.6 points (25.1% vs 39.7%). That is a composition difference rather than a bias signal, and it is the baseline the rest is read against.

Against that baseline only one pair carries a self-targeting mark, and it fades. GPT-5.4 + Codex, the pair its 63 tasks were calibrated against, gaps 28.9 points where the board averages 14.6, while Claude Opus 4.6 + Terminus-2 gaps 13.4, slightly below the board. Under Codex that mark then shrinks from 28.9 points (GPT-5.4) to 22.8 (GPT-5.5) to 14.3 (GPT-5.6 Sol), and the lighter bars make the endpoint explicit: the current generation of each calibration family sits at the board average, so neither is measurably disadvantaged on the set built from its ancestor's failures.

Semantic perturbation: does the difficulty survive rewording

Quirk exploitation calls for a more direct attack: if a task's difficulty lives in its phrasing rather than its work, rewording the instruction should move the pass rate. So we ran a paraphrase fuzzer against the frozen set. Gemini 3.5 Flash rewrites an instruction as aggressively as it can (sentence structure, voice, vocabulary, paragraph organization) while a validator forces every file path, identifier, number, and enumerated requirement to survive verbatim; the environment and verifier stay untouched. We drew 20 tasks from each calibration subset and re-ran each reworded task under the board protocol with its own calibration pair, GPT-5.4 + Codex on the GPT subset and Claude Opus 4.6 + Terminus-2 on the Opus subset.

20 tasks per subset, before and after paraphrase, 5 trials per task per condition with the calibration pair.

Neither subset moved in a direction the experiment can distinguish from noise: 36.0% to 40.0% on the GPT subset, 54.0% to 48.0% on the Opus subset. The two moved opposite ways, by 4 points up and 6 points down, which is the signature of sampling variation rather than of an effect.

Our reading is that the difficulty is in the work, not in the wording. Two bounds are worth stating. A null on the average does not rule out individual tasks that are phrasing-sensitive; it says there is no systematic effect, not that no task has one. And each task saw a single paraphrase from a single rewriter, so what the experiment establishes is robustness to this kind of rewording rather than to every possible restatement.

What is next

More entries are coming over the next weeks and months, each held to the same standards of deterministic verification, calibrated difficulty, uncertainty-aware reporting, and held-out tasks. We have also started on Terminal Tasks v2, calibrated against the current frontier the same way v1.0 was calibrated against the frontier of its day.

We will keep running v1.0 against new models as they ship and publish each result. Today the best configuration completes just over three attempts in five, and the hardest tasks on the set are still solved on fewer than one attempt in twenty.

Questions and evaluation requests: