SlopCode: Vetto’s version

Leaderboard

Most coding benchmarks evaluate a single assignment: the agent may take many steps inside one session, but it delivers one final answer to one self-contained task. Real software is not built that way. Requirements arrive one at a time, each change lands on code that an earlier session left behind, and both the work and the mistakes accumulate. SlopCode Vetted is built around that. Each task is one project developed through a sequence of feature requests, the checkpoints. Every checkpoint is a new agent session that inherits the repository but not the conversation, and after it a hidden test suite runs with the new tests plus every earlier one; a checkpoint passes only when the whole suite passes.

We ran 6 models on 50 tasks, three times each; one complete attempt through every checkpoint is a run. The report reads the results in two parts. First, how the models do on the tests: the leaderboard below, then correctness along the way and what it costs. Then, the quality of the code they produce: how it grows, how complex it gets, and how far it is from the code people wrote for the same software.

SlopCode Vetted model results

Models are ranked by the share of runs whose last checkpoint passes every test. A run can finish near 100% of tests passing while having failed complete checkpoints along the way.

The models end far apart on a fully passing project: 83.7% of Sol 6.1’s runs finish with every test passing, against 23.6% for Grok 4.7. Yet every model still passes most individual tests at the end, from 93.2% to 99.8%. The failures are few at any one moment, but they are spread over many checkpoints and often carried from one to the next. That gap is the cost of building a project one request at a time, and the rest of this report follows it: first in correctness along the way, then in the code each model leaves behind.

The task suite

The idea comes from SlopCodeBench, the original benchmark of coding agents under iterative, multi-checkpoint development. We built our own version of it: 50 tasks derived from open-source projects, each split into a sequence of checkpoints, run on 6 models with three runs per task and model.

Each task is derived from a real open-source project: a library or command-line tool whose behavior is re-specified as a sequence of feature requests. The agent never sees the original code. It receives a runnable workspace, one request per checkpoint, and is graded by hidden tests written against the original project’s behavior. Every original implementation passes every graded scenario.

The projects span parsers and serializers (YAML, CSV, XPath, Markdown, HL7, AIS radio messages), configuration and policy engines (Kconfig, HCL, IAM policy simulation, Helm values), interpreters and template engines, version and dependency tooling, and protocol encoders. Tasks range from 5 to 12 checkpoints (368 in total); the hidden suite at the last checkpoint ranges from 64 to 229 tests (median 126).

The two things we evaluate

• Does it work? Strict checkpoint success, the final test-pass rate, regressions carried forward, and the cost of getting there.
• What code is left behind? Size and growth; debt, complexity, smells and duplication; function and file shape; error masking; and the distance from the human project.

Correctness along the way

A final snapshot hides how a run got there. This section follows correctness checkpoint by checkpoint: how failures accumulate as requirements are added, and whether behavior that once worked stays working.

Failures accumulate differently

Tasks have between 5 and 12 checkpoints, so we align them by position: the first checkpoint, the checkpoints a quarter, half and three quarters of the way through, and the last one. The value is the share of that checkpoint’s hidden tests that fail.

Share of hidden tests failing, over the course of a task. Mean over tasks, by position within the task. Hover a position for every model’s value.

Why does the curve sometimes dip instead of only rising? Not because models go back and repair what they broke: once a hidden test fails, it almost never passes again in the same run. Of the 3,244 tests that started failing at some checkpoint before the last one, 86% were still failing at the end, and the share later fixed ranges from 5% (Sol 6.1) to 19% (Grok 4.7). The dips come from the denominator: every checkpoint adds new tests, so the same failures become a smaller share of a larger suite, and each position mixes tasks of different lengths.

Regressions are carried forward

A carried regression is earlier behavior that passed, later broke, and was still broken at a subsequent checkpoint. Because each session starts without the previous conversation, a break that the next session does not notice stays in the repository.

Why runs fail

Failure cases were reviewed across the 50-task suite, test by test, and each mechanism was assigned to one category. The numeric breakdown below still reflects the original 37-task set of classified runs. Counts below are failure occurrences (one mechanism in one run), not failing tests: a single missed rule can fail dozens of tests. Models with few failures have few occurrences, so their mix is less stable.

Failure mechanisms by model. Share of each model’s failure occurrences, on the original 37-task classified subset. Totals per model: Grok 4.7 152, Muse 1.3 111, Gemini 3.8 51, Kimi K3 33, Opus 5.5 18, Sol 6.1 7. Hover a cell for its count.

The most common mechanism overall is “wrong rule”. A few representative mechanisms, described without naming the model:

• Output detail · cron_dst. The spec gave a full expression and its full description with all day fields wildcard and no trailing every day. The models appended every day to span descriptions whenever the hour field was restricted, contradicting the delivered example word for word.
• Regression · celpy. A c1 rule says membership with a comparison that raises is an evaluation error. A c2 sentence says a type descriptor is not a string so equality with a string is false. Two trials rewrote all cross-type equality to return false, so the c1 error cases silently returned false from c2 on.
• Missing requirement · ais_aivdm. Queue entries must carry the framing fields plus the full decode field set, including the message type. Two trials deliberately dropped the message type from each entry because the framing attribute repeated it, against the sentence that fixes the entry’s field set.
• Error handling · econf. In strict mode a configuration line holding a bare property name with no separator should be a fatal syntax error. Several models had written their parser at the first checkpoint to silently skip such lines, carried it through the strict-mode checkpoint, and returned exit code 0.
• Interface contract · xpath. The response envelope has a result field that holds the evaluated value directly. Three trials built a response helper that already wrapped its argument and then passed an object with its own result key, so every value came back as nested result objects from c2 to the last checkpoint: respond("ok", {"result": ...}).
• Unrequested behavior · earley_costs. A grammar whose second rule is never referenced was expected to parse. The models rejected it with an accessibility check on by default, a check the text only introduces later as an opt-in strict mode, apparently reproducing the real library’s validation. Once the opt-in mode arrived they flipped the default and the test passed.

What it costs to get there

Every run used the same minimal agent scaffold, so differences in cost reflect each model’s price and how much work it does per request: how many steps it takes, how many tokens it reads and writes. The most expensive model costs 10× as much per run as the cheapest, and cost is not ordered like score.

Score versus cost. Mean platform-billed cost of one run, all checkpoints included, on a log scale. Up and to the left is better. Hover a point for its model.

Cost per passed checkpoint. Total spend divided by the checkpoints passed outright, in US dollars.

Agent steps. Median model calls in one run.

The cheapest model per run, GPT 6.1 Sol, is also best on runs that end with every test passing. Steps and output volume vary several-fold between models: the median run takes from 116 to 648 agent steps.

Does the code get worse as it grows?

This is the central analysis in SlopCode Vetted. Test rewards tell us whether behavior works. They do not tell us whether the implementation is small, legible, modular, or easy to change at the next checkpoint.

Question 01: does quality deteriorate per line?

Raw totals rise whenever a repository gets larger, so they cannot by themselves show deterioration. For each metric we compare its density (per 1,000 lines) at each run’s first and last checkpoint, and set the metric’s total growth next to the code’s own growth. If the two grow at the same rate, the code got bigger without getting worse per line; if the metric outgrows the code, each new line carries more of it.

Quality per line, metric by metric. Median over tasks, at the first and last checkpoint of each run. Pick a metric.

What this shows

Every codebase grows several-fold over a task (Question 02 shows by how much), and the totals grow with it: median technical debt rises 3.1–6.4× between the first and last checkpoint, while the code itself grows 4.0–6.1× in the same snapshots. For 3 of the six models debt grows within 10% of the code’s own growth, and for the rest the two differ by at most 23%: a large part of the extra maintenance burden is simply more code. Per 1,000 lines, 4 of the six models end the run without any measure rising by more than 10%. Where density does rise, it rises most often in technical debt and code smells.

What separates the models is level, not trend. Muse 1.3 carries 656 minutes of debt per 1,000 lines at the end, Kimi K3 220, and those positions are already visible at the first checkpoint. Sequential delivery mostly multiplies whatever style a model starts with. Question 03 looks at where it collects: per-line averages can stay flat while complexity gathers in a few functions.

Question 02: how much does the codebase grow?

Lines of code here means non-blank lines in the agent’s own implementation files; agent-written tests and vendored or generated code are excluded. More code is not automatically worse, but every additional line expands the surface future sessions must understand and preserve.

Implementation size. Median over tasks, in lines of code.

Growth relative to the first checkpoint. Median over tasks. 1.0× is the size of the model’s own first checkpoint: a model that starts small can grow fast and still end small.

What this shows

Sol 6.1 solves the same tasks with the least code: its median implementation ends at 2,274 lines, against 5,094 for Muse 1.3, and it also starts smallest (514 lines at the first checkpoint). It is also the model with the highest checkpoint rate, so less code is not bought with less behavior. Opus 5.5 grows least in relative terms (2.8× its first checkpoint), but largely because it starts with the most code (967 lines): it front-loads structure rather than adding less. Muse 1.3 grows the most, 5.8×.

Every model multiplies its codebase several times over a task, and most of that growth happens after the first checkpoint: the first session writes between 514 and 967 lines, the last ends between 2,274 and 5,094. Growth is expected, since every checkpoint adds requirements. This growth is why Question 01 reads quality per line rather than in totals.

Question 03: where does complexity accumulate?

Per-line density can stay flat while logic becomes concentrated in a few difficult functions. We therefore look at cognitive complexity three ways, at each run’s first and last checkpoint: the total, the typical function, and the share of functions above the hard-to-follow threshold.

What concentration looks like in code: the same rules for reading a key=value line, written two ways. Numbers in the margin are each line’s contribution to cognitive complexity; an if costs more the deeper it is nested.

def parse(line):
  if line:                        # +1
      if line.startswith("#"):    # +2
          return None
      else:                       # +1
          if "=" in line:         # +3
              k, v = line.split("=", 1)
              if k.strip():       # +4
                  if v.isdigit(): # +5
                      return k, int(v)
                  else:           # +1
                      return k, v
              else:               # +1
                  raise ValueError
          else:                   # +1
              raise ValueError
  return None

def parse(line):
  if not line or line.startswith("#"):  # +2
      return None
  k, v = split_pair(line)
  return k, coerce(v)

def split_pair(line):
  if "=" not in line:   # +1
      raise ValueError
  k, v = line.split("=", 1)
  if not k.strip():     # +1
      raise ValueError
  return k, v

def coerce(v):
  return int(v) if v.isdigit() else v  # +1

A toy example written for this report, not code from a run. Both versions apply the same checks; the first stacks them inside each other, so its single function scores 19 and lands above the hard-to-follow threshold of 15, while no function in the second scores more than 2.

What this shows

Total cognitive complexity grows with the code, as Question 01 would predict. The typical function changes much less: it gets more than 10% harder for Opus 5.5, Muse 1.3, and easier for Sol 6.1, Gemini 3.8. Muse 1.3 is the outlier: its median function ends at 19.8, against at most 8.3 for every other model, and 24% of its functions are above the hard-to-follow threshold.

How far is agent-written code from human code?

Every task is derived from a real open-source project, and every original implementation passes every graded scenario. That gives a human reference for the same behavior. Most tasks cross languages (the original is Go, TypeScript, C or JavaScript while the agents mostly write Python), so the main comparison is about shape: how long and how branchy functions are, how bad the worst function is, how much logic sits in already-complex functions, and how code is split into files. Debt and complexity are compared per line, on every task where the human project was analyzed.

Function shape relative to the human project. Model ÷ human, geometric mean over tasks (47 for Sol 6.1, 50 for the others); 1× is the human project, above 1× is longer or branchier. 6 of 6 models branch more than the human code on the median, the 90th percentile and the worst function alike. Opus 5.5 is closest on cyclomatic p90, Muse 1.3 farthest.

Model functions are about as long as human ones; what differs is how much branching they carry and how the hardest functions look. The tabs below break the comparison out dimension by dimension, with absolute values for every model next to the human median.

Agent code against the human project, dimension by dimension. Median over tasks, at the last checkpoint.

Reading the comparison together

The models are not far from human code in how long their functions are: the median function is 0.9–1.8× the human length. They are far in how much each function decides. The median function branches 1.4–2.8× as much as the human one, the single worst function is 1.3–3.8× as complex, and every model places a larger share of its complexity inside already-complex functions (43–76%, against 29% for the human projects). Agent code is mostly not much longer; it is denser in decisions, and those decisions concentrate.

No model is closest on everything. Opus 5.5 is closest on long-tail function length (p90), median branching, long-tail branching (p90), worst function, complexity inside complex functions, largest-file share; Sol 6.1 is closest on median function length; Kimi K3 is closest on debt per line. Muse 1.3 is the farthest from the human code on most of these dimensions.

File layout is where the models split most sharply. Human projects spread their code over many files (median 42 on the scoped tasks, largest file 12% of the code); Muse 1.3 keeps 93%, Grok 4.7 keeps 73%, Kimi K3 keeps 72% of its code in the largest file. A single large file is not wrong by itself, but combined with branch-heavy functions it is where every later session has to work.

What forms does slop take?

Aggregate metrics describe the result; run readings show how it happens. The examples below are mechanisms observed in real runs, not claims that every occurrence is harmful or that one example estimates a model-wide rate.

Error masking turns invalid input into silent behavior

Defensive code and error masking count broad exception handlers, errors that are silently swallowed, and type guards that return a default or skip input. A type check used only to dispatch between valid formats is not counted. Measured on the 38 tasks where the agents write Python.

try:
  raw = json.loads(raw)
except (ValueError, TypeError):
  raise DomainError("policy.errors")
if validate and not valid:
  raise DomainError("policy.errors")

try:
  doc = json.loads(doc)
except Exception:
  continue
if isinstance(doc, dict):
  docs.append(doc)

Real functions from one task, anonymized by model. The first converts malformed input into an explicit domain error. The second catches any exception and silently skips the input.

Muse 1.3 stands apart on masking density: 36.1 broad or silent handlers per 1,000 lines, against 1.5–6.2 for the other models.

Static-analysis findings by kind

The same rule findings that make up technical debt, grouped by kind and normalized per 1,000 lines at the last checkpoint.

The widest spread between models is in error handling findings (17.0× between the lowest and highest model).

What the comments say

Comments are where a model explains its code to the next session. We drew a fixed random sample of 100 comments per model from the final workspaces and gave each one a primary theme, from content-free narration and section banners to API contracts and the rationale for an edge case.

Content-free comments (trivial narration plus section banners) make up 66% of the sample for Gemini 3.8 and 4% for Sol 6.1. Comments that carry information a reader could not get from the code itself (contracts, edge-case rationale, design and algorithm explanations) range from 25% to 95% across models. The corpus counts in the table also differ widely: from 11,917 to 48,009 comments in the final workspaces, agent tests included.

Architecture drifts when old and new patterns remain live

Architecture drift means a checkpoint adds a second parser, validator, registry, or equivalent pattern while leaving the earlier implementation active. Because this requires understanding intent rather than matching syntax, it is an exploratory LLM-as-a-Judge study: one run per task and model in the original 10-task analysis subset, every transition read blind to the model with the full diff, all by the same reader.

Transitions that add a second live pattern. AI-read, on the original 10-task analysis subset, six models, clear cases only: a duplicated role of meaningful size with both copies live. Whiskers are the intervals.

Every model leaves a second live pattern behind in a sizeable share of its transitions, from 23% (Opus 5.5) to 64% (Muse 1.3) of transitions in clear cases. Counting borderline small copies as well, every model lands between 57% and 79% and the intervals overlap, so the ordering rests on the clear cases. With ten runs per model, read this as a direction, not a ranking: only the two ends are separated by their intervals.

Incremental versus all-at-once

As a control, we also gave each model the full sequence of requirements in one prompt. The results displayed here are from the earlier control cohort, not a recomputed 50-task comparison. Within that cohort, the share of runs that end with every test passing stays in the same range, but the resulting programs are much shorter. This isolates one cost of sequential work: code and decisions accumulate across sessions.

Reduction in lines of code versus incremental. Final workspace, tests excluded. Earlier control cohort.

All-at-once has a single checkpoint, so the common correctness view is whether the final workspace passes every test. The incremental column is the leaderboard’s last checkpoint fully passed rate.

Shorter does not mean cleaner. In all-at-once runs, Sol 6.1 has +35% debt, +32% cyclomatic complexity, +42% cognitive complexity and +40% smells per 1,000 lines. The control points to a tradeoff: incremental delivery encourages accumulation, while one large request compresses similar behavior into fewer, denser structures.

Model profiles

Each profile lists the dimensions on which a model ranks best or weakest among the six; the table after them gives every model’s rank on every dimension (1 = best; for cost, erosion, file concentration and masking, lower is better).

GPT 6.1 Sol

Best on last checkpoint fully passed (83.7%); mean checkpoint rate (90.1%); final tests passed (99.8%); cost per run ($2.49); regression-free runs (87.5%); consistency, all 3 runs perfect (72.9%); broad or silent handlers (1.5 per 1,000 lines).

Claude Opus 5.5

Best on erosion (43%); largest-file share (24%).

Kimi K3

Never best or weakest on these dimensions; mid-table throughout.

Gemini 3.8 Flash

Weakest on cost per run ($25.02).

Muse Spark 1.3

Weakest on erosion (76%); largest-file share (93%); broad or silent handlers (36.1 per 1,000 lines).

Grok 4.7

Weakest on last checkpoint fully passed (23.6%); mean checkpoint rate (47.6%); final tests passed (93.2%); regression-free runs (23.6%); consistency, all 3 runs perfect (10.4%).

Each cell is the model’s rank among the six, then its value.

Why these findings matter

Correctness remains the first requirement for a coding agent. SlopCode Vetted shows why it is not a complete training signal. Two runs can receive almost the same final test reward while creating very different future work: one leaves localized decisions; another leaves a monolith with broad exception handling, parallel implementations, and behavior outside the request.

This matters for long-horizon agents and reinforcement-learning environments. A reward attached only to the current checkpoint can make locally useful patches rational even when they increase the cost of every later request. Sequential requirements, regression tests, and measurements of the evolving workspace expose that delayed cost.

The results also argue against collapsing a familiar static-analysis total into a quality score. Totals mostly rank how much code a model writes. Density removes part of that confound but can reward compact, highly concentrated logic. Function shape reveals concentration but cannot tell whether the code implements more behavior than requested. Slop is not one existing metric with a new name.

What’s next

We are exploring whether these signals can be compiled into a smaller set of benchmark metrics. The current evidence suggests separating structural slop—concentration of decisions, duplication, error masking, and architecture drift—from excess relative to the request, which requires the specification and a view of how much code or behavior was added for each graded requirement.

We have not selected or validated a combined score. Early candidates were vulnerable to language differences, test coverage, or trivial dilution. We plan to validate the surviving ideas on held-out, same-language tasks and report them as a follow-up rather than presenting an unfinished scalar metric as a result of this release.

Limitations

• Three trials per cell. 50 tasks and three runs per task and model make aggregate gaps of a few points directional, and single task–model cells indicative only: one run moves a cell by a third of its range.
• Language mismatch. Most human references are written in a different language from the agents’ solutions, so size and rule-density comparisons with human code are avoided; the human comparison uses function and file shape.
• Static metrics are not correctness metrics. Tests measure behavior; static analysis describes the implementation left behind.

Appendix

Task set and protocol

The benchmark covers 50 tasks derived from open-source projects originally implemented in Go, Python, TypeScript, C, JavaScript. Each task contains successive feature requests, a runnable workspace, and hidden tests attached to each checkpoint. Every checkpoint is a fresh agent session on the inherited workspace. All models use the same minimal agent scaffold and limits. Results are computed per run, averaged within a task, then averaged across tasks so long tasks do not automatically dominate the aggregate.

Metric definitions

Mean checkpoint rate is the fraction of checkpoints whose complete cumulative test group passes. Final test pass is the fraction of individual tests passing in the last workspace. Last checkpoint fully passed is the share of runs whose last checkpoint passes every test. Regression carried counts earlier-green behavior that fails later and remains broken. Cost is the platform-billed cost of one run; cost per passed checkpoint divides a model’s total cost by the checkpoints it passed outright.

Lines of code are non-blank lines of the agent’s implementation files, excluding agent-written tests, vendored libraries, binaries and generated files. Technical debt sums fixed repair-time estimates for static-analysis findings. Cyclomatic complexity counts independent control-flow paths. Cognitive complexity adds penalties for nested and hard-to-follow flow. Erosion is the share of total cyclomatic complexity inside functions with cyclomatic complexity above ten. p90 is the value below which 90% of functions fall. Ratios to the human project are geometric means over tasks.

Code smells are maintainability findings such as excessive parameters, repeated literals, or empty branches. Duplicated lines are lines inside repeated blocks of at least six normalized lines. Defensive code covers broad exception handlers, silent swallows, and type or attribute guards that default, skip, or raise rather than merely dispatch between supported input shapes.