ProgramBench Vetted

ProgramBench Vetted

Can an agent rebuild a program from a runnable binary? That is the fundamental question behind ProgramBench. We are releasing ProgramBench Vetted, a set of 50 tasks built around the same premise, with improvements to the task design process that make the benchmark fairer and more reliable.

In addition to preserving measurable partial progress and deterministic test based grading, our task design process adds a broader set of controls for failure modes involving test duplication, environment quality, hackability, and fairness.

ProgramBench matters because program reconstruction is one of the relatively few coding benchmark formats built around very long-horizon work. An agent must preserve and revise a model of the program across an extended trajectory, which makes these tasks useful for studying long-context coordination. They are also valuable for RLVR: many deterministic behavioral checks can provide dense, behavior-driven reward while the agent still has to integrate that feedback into one coherent implementation. Because these tasks come from public repositories, some may have appeared in the pretraining data of large models. Yet prior exposure does not imply that a model can retrieve or reconstruct the implementation from memory when asked to move from observed behavior back to code. The Reversal Curse offers a useful analogy for this gap. We discuss the resulting memorization paradox and its training implications in Appendix D.

What ProgramBench is

ProgramBench, released by Meta in May 2026, gives an agent a compiled program it can execute but not read, together with its usage documentation. The agent must write a new codebase that reproduces the observed behavior.

Each of the 200 tasks is a real open source CLI project, from small utilities up to FFmpeg, SQLite, and the PHP interpreter. The intended task package is a sealed container: the binary is available only for execution, the documentation is kept, and the source and git history are removed. The agent gets a shell, no internet, a thousand steps, and six hours. Behavioral tests grade the result. The tests were themselves written by an agent that probed the original repository and binary.

The original construction process has three stages:

1. Build the reference. Select a repository in a compiled language and have a build agent produce the gold binary.
2. Generate behavioral tests. A test generation agent explores the repository and binary with access to the source, existing tests, and documentation. It can adapt upstream tests and uses source line coverage to guide further exploration.
3. Validate and seal. Every generated test must pass an assertion linter, pass against the gold binary, and fail against a dummy binary. A separate step then reduces the model's workspace to the executable, documentation, and retained test assets.

At evaluation time, the source, repository tests, and git history are absent. The agent keeps the documentation and can execute the reference binary while building its replacement. The generated behavioral suite then grades the reconstruction by the fraction of tests passed; a task is resolved only when every test passes.

The retained documentation provides part of the specification, but not the complete behavioral contract. The rest must be inferred by probing the binary, forming a hypothesis, implementing it, comparing the result, and revising. The four frontier agents analyzed in the paper spend 23 to 34% of their actions probing the reference, making empirical investigation a substantial part of the task rather than a brief setup step.

Why ProgramBench Vetted is different

ProgramBench Vetted is not designed simply to be harder. It is designed so that each point of reward more faithfully represents the intended behavior. Our target is calibrated difficulty: a task should leave room for meaningful partial progress, but should not award most of its score for reproducing a shallow interface or exploiting a defect in the evaluation.

Our internal ProgramBench generation pipeline

Like the original ProgramBench, our pipeline turns a repository into two coupled artifacts: a cleanroom reconstruction environment and a behavioral test suite that defines its reward. To address the failure modes we found while auditing the released dataset, however, we added validation and repair throughout construction rather than relying on a single final review.

Continuous review by researchers provides the seed for specialized Agents as Judges and Agents as Doctors. The judges investigate the workspace, test the relevant behavior, and produce scored reports and evidence for review. When a promising task has a repairable problem, the doctors use that evidence to make a constrained change before the task passes through the same checks again. These loops operate alongside deterministic gates for properties that can be measured exactly and adversarial agents that actively try to exploit the task or its reward. Human reviewers handle ambiguous cases, make the final decisions, and turn recurring failure modes into stronger criteria for later reviews. We distribute compute across these specialized stages because build integrity, environmental leaks, test quality, and reward exploits require different forms of investigation; no single score can cover them all. This combination of specialized compute and accumulated review knowledge is central to how we build the tasks. The four exhibits below divide the process into connected stages with many steps.

The current candidate set

The candidate set contains 50 held out, verifier graded tasks. Construction uses five frontier models across up to 15 independent test generation trials per task, coverage guided exploration, structural deduplication, leakage checks, and adversarial review:

1. Shortcut checks. The pipeline probes named failure classes: non varying gold behavior, reachable reference implementations, constant pinning tests, and degenerate suite shapes.
2. Calibration. We run multiple models against each candidate and study the distribution of partial credit. When an observed score does not reflect the behavior the task is meant to measure, we inspect why, revise the tests, documentation, or environment, and run the task again. The goal is a useful difficulty range: enough accessible behavior to measure progress, enough unsolved behavior to preserve headroom, and a score that changes as the quality of the reconstruction improves. Tasks that collapse into almost uniform success, almost uniform failure, or a sharp all or nothing threshold receive further review.
3. Fairness and hackability. We investigate both requirements that a genuine implementation cannot satisfy and shortcuts that earn credit without rebuilding the intended behavior. A fair task makes specified behavior reachable; a robust task does not expose a cheaper path to its reward. In the original dataset, FFmpeg depends on absent reference inputs (example), a documentation echo clears 79.5% of the cmatrix suite without implementing its core behavior (example), and an eight-line wrapper around the XZ executable already installed in the task environment earns 77.3% of its active reward (example). We encode lessons from cases like these as construction gates and repairs rather than treating difficulty alone as evidence of quality. Questionable cases are replayed against the reference program and the sealed environment, with the tests, documentation, runtime image, and task contract reviewed as one system. When the evidence points to an unreachable requirement, an underspecified interface, or a shortcut prone reward, we revise the relevant tests, documentation, or environment and run the task again.
4. Deduplication. We consolidate pairs classified as covering the same or a subsumed behavior.

Leaderboard comparison

Metrics. Mean reward is the macro average of each task's fractional test score. Almost is the share of tasks reaching at least 95%; resolved is the share reaching 100%. All ProgramBench Vetted results use pass@1. A hyphen indicates that no result is available. * vendor reported; follow the star for the source.

These columns separate broad partial progress from end to end completion. GPT 5.5 leads ProgramBench Vetted in mean reward at 85.2%, but resolves 2% of tasks; Claude Opus 5 has a slightly lower mean reward at 81.9%, yet leads both almost at 50% and resolved at 14%. Six of the 17 evaluated models resolve at least one task. Among configurations reported on both benchmarks, the direction of the change is mixed rather than uniform, so the table does not support treating ProgramBench Vetted as simply easier or harder.

Exhibit 6 compares how partial credit is distributed across ProgramBench's current leaderboard and our complete ProgramBench Vetted pass@1 evaluations. For each model, the curves show the share of tasks cleared at every fractional test score threshold. In the Vetted panel, the curves remain visibly separated across much of the threshold range while retaining meaningful partial credit below full resolution. We interpret this wider separation as evidence that the dataset preserves more of the performance differences among current models instead of compressing them into the same score band.

The curves do not establish that either benchmark is intrinsically harder. The panels evaluate materially different model sets, only a subset of configurations overlaps exactly, the original includes smaller models such as GPT 5 mini, and several recent frontier configurations were evaluated only on ProgramBench Vetted.

Fractional test score survival curves. The left panel redraws ProgramBench's current extended leaderboard because the paper's figure predates the latest results; the right applies the same threshold sweep to our complete pass@1 evaluations. In ProgramBench Vetted, submissions that do not compile or run receive a score of zero; the original panel reproduces the leaderboard's published curves. * updated manually from the leaderboard in programbench.com

Task level difficulty distribution

This view groups tasks by their mean fractional test score across the complete model panel for each benchmark. ProgramBench Vetted deliberately contains fewer tasks at the near-impossible end of the distribution. In the model panel shown here, 46 of 50 tasks fall between 20% and 80% mean reward. This calibrated range preserves room for meaningful partial progress while leaving enough behavior unsolved for an improved model to turn progress into a resolved task. It also avoids concentrating the benchmark on tasks that provide almost no learning or evaluation signal. Because the two panels evaluate different models, the comparison remains descriptive.

Distribution of mean fractional test scores by task. Each task is averaged over the complete model evaluations available for its benchmark, then assigned to a ten percentage point bin. Bars are normalized within each benchmark.

Average steps per task

For every model configuration evaluated on both benchmarks, ProgramBench Vetted requires more model calls per task on average. The comparison shows that calibrated tasks drawn from smaller or less familiar repositories can still sustain longer solution trajectories than the original task set, without depending on unusually large projects such as FFmpeg.

What kinds of programs

Every task is a behaviorally testable CLI program, but that spans very different kinds of work. We classify candidates into a fixed taxonomy so the release does not collapse into one kind of CLI. The archetype is a language implementation: an interpreter, compiler, or bytecode VM, because its input space is unbounded and cheap to explore: every new program as input is another behavior to pin.

The original dataset and why build a new one

We began by treating ProgramBench as a recipe for a larger benchmark and training corpus. Rebuilding that recipe at scale exposed two publishable findings: duplicated active reward in a purposive audit, and concrete active credit earned without rebuilding the intended program. The claims below are limited to the released artifacts and executed cases we could reconcile.

Dataset scope. Every named task in this section, including cmatrix, nnn, xq, ripgrep, LZ4, QuickJS, htop, and XZ, is a task from Meta's original ProgramBench release. Where we ran a new adversarial candidate, we say so explicitly; those candidates and trajectories are ours, while the task images, test manifests, and scored active suites are the original dataset's.

Failure screen methodology. As we rebuilt ProgramBench, our reviewers turned recurring failure patterns into a set of guidelines and gates applied across the task suites. A gate flag starts a targeted adversarial review: specialized agents attempt to exploit the suspected shortcut or reproduce the suspected contract failure. We then reconcile their findings with the active manifest, separate execution failures from test failures, and inspect the relevant tests, inputs, and reference binary behavior. The cases below are confirmed findings produced by that process.

1 · Duplicated active reward in a purposive audit

The released manifests contain 247,723 active test IDs across 1,759 test generation runs, which the dataset calls branches.

Two tests count as duplicates when they exercise the same or materially similar behavior, or when the behavior checked by one is subsumed by the other. The analysis includes exact and close duplicates within and across generation trials, with code level review of every candidate pair. Readers can inspect matched test bodies directly by clicking points in Exhibit 10. Across the audited set, the median task carries 11% of its scored suite as removable duplicates of another scored test; more than half the sixty audited tasks are above a tenth, and the highest measured share is 37% (Exhibit 9).

Our construction also uses multiple generation trials, so we added a deduplication stage that consolidates tests classified as exercising the same or subsumed behavior. The recurring patterns we found also changed the harness itself: deterministic gates catch exact and structural repetition, stochastic agent based gates examine behavioral similarity and subsumption, and human review resolves ambiguous cases and feeds new patterns back into those checks. The interactive view below shows the linked test bodies available for direct inspection.

QuickJS provides a useful comparison. Its 3,034 scored tests have a 6.1% duplicate share in our internal audit, about one sixth of tui journal's rate. Most clusters in its map represent genuinely different JavaScript behaviors. Tests may appear close because they share code structure, so proximity alone is not enough to classify a duplicate.

Duplication matters beyond suite cleanliness. When several scored tests reward the same basic behavior, that behavior receives several times the weight of a distinct requirement. At evaluation time, this can favor models that repeatedly clear a narrow family of similar checks. At training time, the same imbalance can steer reinforcement learning toward the most repeated behaviors rather than the full program contract. At scale, the next question is therefore not only whether a test is valid, but whether its reward can be earned too cheaply.

2 · The reward can be bought

The severe one. The original validates every test two ways: must pass the gold, must fail a dummy. Both are necessary. Neither is sufficient. Our audit of the released original tasks found cases where a submission that provably does not rebuild the program still collects active credit.

The common shape is a model writing tests from observed behavior against programs whose defining behavior can be hard to observe through stdout and an exit code. A gold pass and dummy fail screen does not ask whether a surviving test's reward was earned by rebuilding the target. The original release contains several concrete ways around that distinction.

• Score inflation: cmatrix. Several active tests ask whether a flag is accepted but combine it with -V. The program exits after printing its version, before the flag's behavior can run. A small program that only reproduces the help and version text, without rendering the matrix animation, passes 79.5% of the active suite.
• Withheld credit: FFmpeg. Of 1,091 active FATEWhat is FATE? FATE, the FFmpeg Automated Testing Environment, is FFmpeg's regression suite. It exercises codecs, formats, and filters against reference media and expected outputs. cases, 1,029 require media files missing from the released archive and images. Both inspected gold builds fail all 1,029. A normal implementation cannot process an input the benchmark did not ship, so these cases measure packaging failure rather than reconstruction ability.
• Ambient implementation access: XZ and LZ4. The original task environments expose working implementations of the programs being reconstructed. An eight-line wrapper around the installed XZ executable earns 77.3% of its active reward. In LZ4, delegating compression to the installed library raises the candidate's score by 31.62 percentage points. In both cases, part of the measured reward comes from reusing an implementation already present in the evaluation environment.

These cases establish the three mechanisms above; the executed evidence is collected in the appendices, with the test identities, environments, reference replays, and causal controls used for each finding.

Why these findings matter for reward design

The same test suite defect has different consequences in evaluation and training. In an evaluation it can distort a ranking. If a similar suite of RLVR environments is used for reinforcement learning, it also becomes part of the objective. This section states design implications, not results from a training run.

For evaluation, a weak suite can misweight scores. If a version banner check and a real behavioral test each count for one point, a model that only learned to print the banner gets the same credit on those two tests as one that implemented the substantive behavior. Duplication compounds the weighting: a behavior rediscovered by many generation trials is counted many times. That weighting can alter rankings, although we did not run a matched ranking experiment here.

For reinforcement learning, the same suite is both a measurement and an instruction. Test multiplicity and shortcut passable cases determine how much reward is available for each observed behavior. That mathematical weighting is established by the suite; how a particular policy responds would require a training experiment.

• A skill independent floor becomes available reward. The original's dummy pass gate removes many literal no op failures, but residual active credit remains in each of the three original tasks shown in our selected no op audit (Exhibit 15). A small program that only reproduces help and version text, without rendering the matrix animation, passes 79.5% of original cmatrix (Exhibit 12). An eight line wrapper that calls the preinstalled XZ program passes 77.3% of original XZ (Exhibit 13). Both earn substantial reward without implementing the program's core behavior.
• Duplicate tests re weight the objective. A behavior counted many times over is worth that many times the gradient contribution of an otherwise equivalent single test. This does not prove that a policy will specialize in that behavior, but it does prove that the objective weights it more heavily.
• A shortcut creates a cheaper path to reward. Offered a choice between printing a captured --help string and implementing the parser, the suite pays for the string immediately. One scored test, for example, claims to check a color flag but asserts only that the version banner printed (Exhibit 11).

This is why we treat the findings above as engineering constraints rather than footnotes. The shortcut checks, calibration design, and deduplication stage are intended to make the tested shortcuts less rewarding and align more of the score with rebuilding. Whether they improve downstream training is an open empirical question.

The tests do not describe the task. For a model trained against them, the tests are the task.

What's next

Rebuilding programs measures one demanding slice of software ability. Saturating this benchmark would not, by itself, make a model a great engineer. The verifier checks behavioral reproduction, not code quality: a monolithic single file solution that matches the binary scores exactly the same as a well structured one, and speed, memory, and footprint go unmeasured. The task also hands the agent something real engineering almost never has, a perfect oracle for the spec. Everyday work means ambiguous requirements, evolving specs, other people's code, and codebases that live for years; none of that fits inside a rebuild episode. A model could top this leaderboard and still be a poor collaborator. That is why the family has to keep evolving, and why the recipe matters more than any one instance of it.

Growing and recalibrating ProgramBench Vetted

We will continue increasing the number of validated tasks and updating the benchmark as models improve. Each update has three goals:

1. More fair. Broaden the mix of tasks while removing shortcuts and requirements that an implementation cannot reasonably satisfy.
2. More reliable. Strengthen replay, deduplication, environment controls, and ongoing validation so the reward continues to measure the intended behavior.
3. Harder as models evolve. Add more demanding but still reachable behaviors, so stronger models have room to improve and partial progress remains measurable.

Beyond compiled code: rebuild anything packaged

Nothing in the recipe is specific to CLI binaries. The general form: take a packaged artifact whose behavior is observable but whose internals are sealed, keep its documentation, generate a behavioral spec by probing it, guard the reward against shortcuts, and grade rebuilds against that spec. We are currently working on these:

1. Spec2Code benchmarks. We are evolving the lessons from ProgramBench into benchmarks built from both public and private repositories, but with an explicit specification as the model's starting point and no reference binary to probe. The behavioral tests are broader and each one must trace to, and remain coherent with, that specification. We believe this is a cleaner and fairer way to evaluate whether a model can implement specified behavior because project documentation often lacks enough context to reconstruct an entire codebase faithfully.
2. APIs and servers. Seal a running service in a container: endpoints callable, implementation invisible. The agent rebuilds the server from the API docs plus probing, and the verifier replays a request suite against both, checking response parity, status codes, error semantics, and state across sequences of calls. The domain brings its own hack surface: proxying requests to the reference is this family's ambient implementation defect, and the gates must ban it the same way.
3. Libraries and SDKs. Ship a compiled artifact (a wheel with native extensions, a jar, a shared object) without source. The agent reimplements a library or SDK from its documented public interface. It is graded with behavioral tests created by exercising the original implementation through that same interface.
4. Protocol daemons. Given a server speaking a wire protocol, rebuild a compatible implementation. Verification is interoperability itself: the rebuilt server must satisfy the reference client, and vice versa.
5. Data pipelines. A sealed transform (a container that turns inputs into outputs) plus a description of its purpose. The agent rebuilds the transform; the verifier checks output parity on held out inputs, including the edge cases a probing agent discovered.

Each variant needs its own validity work because each creates different ways to buy the reward. The durable research problem is not posing a rebuild challenge; it is making the resulting reward trustworthy at scale.

Appendix A · Executed audits of the original dataset

We ran a broader set of audits across the original dataset and selected five representative cases for this appendix. They illustrate recurring issues we found, including proxy and time-dependent assertions, shallow implementations that still earn substantial credit, software already available in the task environment, and missing test inputs. These findings informed the construction and validation gates in our dataset. Each case below separates what we tested, what we found, why it happened, and its implications for the benchmark.

1. Residual reward for an empty program

What we tested. We compiled the same program, int main(void) { return 0; }, and scored it with the original active suites for nnn, xq, and ripgrep.

What we found. The empty program still passed 23, 19, and 22 active tests, respectively.

Why it happened. These tests accept silence, check an empty result, or verify that nothing happened without first proving that the target behavior ran. We found the same pattern in other tasks; the three examples below are enough to show the mechanism.

Implications. This creates a skill independent reward floor: a program that implements no behavior still earns points. In evaluation it weakens what the score means; in reinforcement learning it assigns reward to silence rather than to reconstructing the program.

2. Assertions that measure a proxy

What we tested. We ran active tests from cmatrix, nnn, tig, and tty-clock against the reference binary and deliberately shallow stubs. For the clock test, we also advanced the container date by one year.

What we found. Version-output stubs passed tests named after color, option, and author behavior. The tty-clock reference binary failed when the year changed.

Why it happened. The proxy tests invoke --version, which exits before the named behavior runs. The clock test hard-codes 2026 instead of validating a date.

Implications. Scores can rise without exercising the behavior named by a test, or fall simply because the calendar changed. Both effects introduce measurement error unrelated to implementation quality and make comparisons less fair.

@pytest.mark.parametrize("color", ["green", "red", "blue", "white", "yellow", "cyan", "magenta", "black"])
def test_color_option_accepted(color):
    """Test that valid color options are accepted."""
    # We can't fully test TUI behavior, but we can verify the flag is accepted
    # Use with -V to exit immediately after processing args
    result = run("-C", color, "-V")
    assert result.returncode == 0
    # Should still print version when combined with -V
    assert b"CMatrix version" in result.stdout

        date_str = date_match.group(1)
        year, month, day = date_str.split('-')
        assert year == '2026', f"Year should be 2026, got {year}"
        assert 1 <= int(month) <= 12, f"Month {month} out of range"

def test_nnn_opts_overridden_by_cli():
    """Test command line options override NNN_OPTS."""
    result = run("-V", env={"NNN_OPTS": "d"})
    assert result.returncode == 0
    # -V should still show version, NNN_OPTS ignored
    assert b"5.2" in result.stdout or b"." in result.stdout
    # Should be version, not help
    assert len(result.stdout.strip()) < 20

3. FFmpeg tests with missing inputs

What we tested. We ran all 1,091 active FATE cases against two shipped FFmpeg reference binaries and checked the same identities across 20 published runs.

What we found. Both reference binaries passed the 62 self-contained cases and failed all 1,029 cases that read from /workspace/fate-suite. No published run passed any of those 1,029 identities.

Why it happened. The expected outputs were published, but the required FATE media corpus was not included in the test archive or any released task image.

Implications. These points are unreachable for an implementation that executes the requested behavior normally. They depress scores, weaken comparisons between attempts, and make evaluator-aware replay more rewarding than genuine reconstruction.

/workspace/executable -nostdin -nostats \
  -i /workspace/fate-suite/012v/sample.avi \
  -pix_fmt yuv422p16le -vf scale -bitexact -f framecrc -

expected file: /workspace/tests/ref/fate/012v  (last CRC: 0xc5439580)
actual result: exit 254, Error opening input file /workspace/fate-suite/012v/sample.avi

4. A non-rendering cmatrix implementation

What we tested. One of our agents built a 99-line program that reproduces documented help, version, usage, and option handling but never renders the matrix animation. We then scored it against the complete original suite.

What we found. It passed 403 of 507 active tests, or 79.5%, while failing all nine tests that require the render loop.

Why it happened. Most points are attached to shallow command-line behavior that can be copied from the documentation; comparatively few require the program's defining behavior.

Implications. A high fractional score can therefore overstate reconstruction quality. The suite makes copying documentation disproportionately valuable while giving the matrix rendering behavior comparatively little weight.

def test_help_exits_immediately():
    result = run("-h")
    assert result.returncode == 0
    assert len(result.stdout) > 100

def test_version_flag_short_circuits_other_validation(args):
    rc, out, err = run(args)
    assert rc == 0
    assert "CMatrix version" in out
    assert err == ""

def test_unknown_option_prints_usage_and_exits_zero(args):
    # args = [--nonexistent] / [-Z]
    rc, out, err = run(args)
    assert rc == 0
    assert_usage(out)
    assert err == ""

def test_color_flag_without_value(self):
    returncode, stdout, stderr = run_cmatrix("-C")
    assert returncode == 0
    assert "Usage:" in (stdout + stderr)

def test_u_valid_values_start_matrix_render_loop(args):
    with pytest.raises(subprocess.TimeoutExpired):
        run(args, timeout=0.3)

5. Cheatable environments: implementations already installed

What we tested. We asked an adversarial agent to use any system CLI or library already installed in two original compression-task images, then scored the retained source with the original active suites.

What we found. An eight-line wrapper around /usr/bin/xz passed 1,090 of 1,410 active XZ tests. A 349-line CLI adapter around liblz4.so.1 passed 557 of 1,496 active LZ4 tests.

Why it happened. The task images exposed implementations of the target algorithms to the submission. The candidate could delegate core work instead of rebuilding it.

Implications. The score reflects software supplied by the environment as well as the candidate's capability. Tasks with an installed implementation become easier through a hidden shortcut, breaking comparability and rewarding delegation rather than reconstruction. In a large RLVR corpus, even a small number of such environments can become high yield shortcuts. Repeated optimization against them risks widespread reward exploitation and capability degeneration, as the model learns to search for installed answers instead of acquiring the intended skills.

Determine which system library or CLI already implements this program's core algorithm.
If one exists, write a thin CLI wrapper that calls that ambient implementation.
Implement none of the algorithm yourself.

$ which xz
/usr/bin/xz

$ xz --version
xz (XZ Utils) 5.2.5
liblzma 5.2.5

#include <unistd.h>
#include <stdio.h>
int main(int argc, char **argv){
    argv[0] = "/usr/bin/xz";
    execv("/usr/bin/xz", argv);
    perror("execv");
    return 127;
}

Determine which system library or CLI already implements this program's core algorithm.
If one exists, write a thin CLI wrapper that calls that ambient implementation.
Implement none of the algorithm yourself.

liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1
liblz4-1 1.9.3

$ cc -O2 -o executable lz4wrap.c -Wl,-l:liblz4.so.1

LZ4F_createCompressionContext(...)
LZ4F_compressBegin(...)
LZ4F_compressUpdate(...)
LZ4F_compressEnd(...)
LZ4F_createDecompressionContext(...)
LZ4F_decompress(...)

Appendix B · Ambient implementation reproductions

We tested whether a candidate could earn credit by calling a compression implementation already installed in the original task environment. We rebuilt each candidate without network access, replayed the active suite, and then removed or disabled the ambient implementation to isolate its effect. These two selected cases isolate the mechanism. Measuring its prevalence across the full release is outside this experiment.

Reading the results. The XZ candidate is almost entirely a shortcut. The LZ4 candidate is substantive, but its score depends materially on the installed library. Both values are fractional active test-pass rates; neither candidate fully solves its task. These reproductions measure test-passing behavior; source-policy acceptance is a separate review.

Appendix C · Runnable but difficult test surfaces

We started with a simple warning sign: an active test that had no pass in the available, eligible public per-test results. That does not prove the test is impossible. It only tells us which tests deserve a closer look. We then checked those tests against the released reference program, or another authentic build of the product.

Result: every test in the five screened sets below has an authentic passing witness. We found no FFmpeg-style broken subset here. What we did find were large compatibility cliffs: groups of tests that require a candidate to reproduce one specific product revision, including its internal formats, exact messages, numerical output, and obscure features.

A passing witness establishes technical passability, not practical recoverability. The reference binary can reveal an expected behavior only after the agent identifies the relevant command and supplies a useful input. We therefore also inspected the cleanroom documentation, binary introspection surfaces, representative fixtures, and several active, non-ignored tests across these five tasks. The question was narrower than whether the answer exists somewhere in the binary: could an agent reasonably discover the scored behavior from the supplied information and available probes within a bounded episode?

The answer differs by task. PHP exposes a large interface but does not identify thousands of historical regression inputs. The paper-era Ctags environment exposed the implementation itself. Cppcheck's central analysis path could not be probed until the grader restored a missing dependency. Pandoc is comparatively well documented and introspectable, but exceptionally broad. GROMACS documents its commands while reserving many representative scientific assets for grading. The examples below show each case directly.

--FILE--
<?php
$input = array('One' => 'un', 'TWO' => 'deux', 'three' => 'trois');
for ($i = -5; $i <= 5; $i += 1) {
    echo "\n-- \$sort argument is $i --\n";
    $temp = $input;
    var_dump(array_change_key_case($temp, $i));
}
?>
--EXPECT--
*** Testing array_change_key_case() : usage variations ***

-- $sort argument is -5 --
array(3) {
  ["ONE"]=> string(2) "un"
  ["TWO"]=> string(4) "deux"
  ["THREE"]=> string(5) "trois"
}

--INI--
opcache.validate_timestamps=1
opcache.revalidate_freq=0
--FILE--
<?php
file_put_contents(__DIR__ . '/gh8846-index.php', <<<'PHP'
<?php
if (!@$_GET['skip']) {
    include __DIR__ . '/gh8846-1.inc';
}
include __DIR__ . '/gh8846-2.inc';
echo "Ok\n";
PHP);

php_cli_server_start('-d opcache.enable=1 -d opcache.enable_cli=1');
echo file_get_contents('http://' . PHP_CLI_SERVER_ADDRESS . '/gh8846-index.php');
echo file_get_contents('http://' . PHP_CLI_SERVER_ADDRESS . '/gh8846-index.php?skip=1');
?>
--EXPECTF--
bool(true)
Fatal error: Cannot redeclare class Foo ...
bool(true)
Ok

$ sha256sum /workspace/executable /workspace/ctags
191707e...066c  /workspace/executable
191707e...066c  /workspace/ctags

// input.cpp
struct alignas( 64 ) AlignasTestStruct
{
};

class alignas( 64 ) AlignasTestStruct2
{
};

# expected.tags
AlignasTestStruct  input.cpp  /^struct alignas( 64 ) AlignasTestStruct$/;"  s  file:
AlignasTestStruct2 input.cpp  /^class alignas( 64 ) AlignasTestStruct2$/;"   c  file:

def test_asciidoc_parsing():
    adoc_source = """\
= Document Title
Author Name <email@example.com>

== Introduction
=== Sub Section
== Chapter 2
"""
    with TempFiles() as tf:
        src = tf.create("test.adoc", adoc_source)
        result = run("--quiet", "--options=NONE", "-o", "-", str(src))
        assert result.returncode == 0
        assert b"Introduction" in result.stdout or b"Chapter 2" in result.stdout

Failed to load library configuration file 'std.cfg'.
Your Cppcheck installation is broken.

def test_arithmetic_overflow_detection(run_binary, create_test_file):
    code = """
void test() {
    int x = 2147483647;  // INT_MAX
    int y = x + 1;
}
"""
    test_file = create_test_file(code)
    result = run_binary(['--enable=warning', '--dump', str(test_file)])
    assert result.returncode == 0
    dump_file = Path(str(test_file) + '.dump')
    assert dump_file.exists()
    content = dump_file.read_text()
    assert len(content) > 100
    assert '<valueflow>' in content

def test_dereference_end_iterator(run_cppcheck, temp_dir):
    code = """
#include <vector>

void test() {
    std::vector<int> vec = {1, 2, 3};
    std::vector<int>::iterator it = vec.end();
    int x = *it;
}
"""
    test_file = Path(temp_dir) / "test.cpp"
    test_file.write_text(code)
    returncode, stdout, stderr = run_cppcheck("--enable=warning", str(test_file))
    assert "derefInvalidIterator" in stderr
    assert "Dereference of an invalid iterator" in stderr
    assert "assigned value is end=0" in stderr

def test_cmd_10127(run_pandoc):
    result = run_pandoc(
        args=['-f', 'rst', '-t', 'native'],
        stdin="""=====  =====  ======
   Inputs     Output
------------  ------
  A      B    A or B
=====  =====  ======
False  False  False
True   False  True
False  True   True
True   True   True
=====  =====  ======"""
    )
    assert result.returncode == 0
    assert result.stdout == '[ Table\n    ( "" , [] , [] )\n ...'

def test_pptx_writer():
    with TempFiles() as tf:
        out_path = Path(tf.tempdir) / "output.pptx"
        result = run(
            "-f", "markdown", "-t", "pptx", "-o", str(out_path),
            stdin="# Slide 1\n\n- Point 1\n- Point 2\n\n# Slide 2\n\nContent\n"
        )
        assert result.returncode == 0
        assert out_path.exists()
        assert out_path.stat().st_size > 0
        with zipfile.ZipFile(out_path) as zf:
            names = zf.namelist()
            assert any("ppt" in name for name in names)
            assert any("slides/slide" in name for name in names)

def test_basic_rotmat_system_group(run_gmx, tmp_path):
    output_file = tmp_path / "test_basic.xvg"
    result = run_gmx("rotmat",
                     "-f", str(COVAR_RESOURCES / "trpcage.xtc"),
                     "-s", str(COVAR_RESOURCES / "trpcage.tpr"),
                     "-o", str(output_file),
                     input_data="System\n")
    assert result.returncode == 0, f"gmx rotmat failed: {result.stderr}"

    output = output_file.read_text()
    normalized = normalize_xvg(output)
    expected = normalize_xvg((RESOURCES / "basic_rotmat.xvg.golden").read_text())
    assert normalized == expected

def test_pdb2gmx_amber03_tip3p(run_gmx, tmp_path):
    result = run_gmx("pdb2gmx",
                     "-f", str(RESOURCES / "dipeptide.pdb"),
                     "-o", str(tmp_path / "out.gro"),
                     "-p", str(tmp_path / "topol.top"),
                     "-ff", "amber03",
                     "-water", "tip3p",
                     cwd=str(tmp_path))
    assert result.returncode == 0, f"pdb2gmx failed: {result.stderr}"

    gro_file = tmp_path / "out.gro"
    top_file = tmp_path / "topol.top"
    posre_file = tmp_path / "posre.itp"

    assert gro_file.exists(), "Output .gro file not created"
    assert top_file.exists(), "Topology .top file not created"
    assert posre_file.exists(), "Position restraints .itp file not created"
    assert gro_file.stat().st_size > 100, "GRO file too small"
    assert top_file.stat().st_size > 100, "Topology .top file too small"
    assert posre_file.stat().st_size > 50, "Position restraints .itp file too small"

    top_content = top_file.read_text()
    assert "amber03.ff/forcefield.itp" in top_content, "Force field not referenced in topology"
    assert "amber03.ff/tip3p.itp" in top_content, "Water model not referenced in topology"
    assert "[ moleculetype ]" in top_content, "No moleculetype section in topology"
    assert "[ atoms ]" in top_content, "No atoms section in topology"
    assert "[ bonds ]" in top_content, "No bonds section in topology"

    gro_content = gro_file.read_text()
    lines = gro_content.strip().split('\n')
    assert len(lines) >= 3, "GRO file too short"
    atom_count = int(lines[1].strip())
    assert atom_count == 20, f"Expected 20 atoms after adding hydrogens, got {atom_count}"

What this evidence supports

Passability and reasonable recoverability are separate gates. An authentic gold witness shows that a configured evaluator can produce a pass. It does not show that the agent was given enough information and probing capability to identify the required behavior within its episode. The five cases do not share one defect: PHP has a sparse specification relative to its historical regression surface; paper-era Ctags exposes the answer; Cppcheck's core oracle is incomplete during inference; Pandoc is well specified but very broad; GROMACS describes its commands without supplying representative scientific probe assets.

The calibration question is how reward is distributed. Exact output and deep compatibility can be legitimate requirements. They become problematic when a large, correlated fraction of reward depends on behavior that is not proportionately discoverable, when the grader has dependencies unavailable to the agent, or when the environment already contains the implementation. Those conditions can obscure useful partial progress and make scores less comparable across tasks.

Evidence boundary. “No eligible public pass” is a screen based on the available official per-test artifacts, not a probability of impossibility. One configured PHP run has no PHP per-test artifact, and seven later GROMACS and Pandoc artifacts omit some active rows. Those rows remain missing rather than being counted as failures.

Appendix D · The memorization paradox

ProgramBench uses familiar public programs, so prior model exposure is plausible but difficult to measure. This appendix separates what the benchmark shows from what remains unknown, then treats the same distinction as a possible training variable.

FFmpeg, SQLite, Lua, and jq are highly visible public codebases. The paper's internet access ablation found up to 36% of runs flagged as cheating, mostly through source code lookup. This shows a strong incentive to retrieve source when it is available, while leaving the contents of model weights unresolved. At the launch evaluation, zero tasks were resolved.

Rebuilding a familiar program may therefore combine partial prior representations with reverse engineering from documentation and binary probes. Matched contamination and name masking ablations would separate those effects. Until then, reconstruction performance is best read as the composite capability measured by the task.

The paradox splits the benchmark's meaning in two. One ablation forces a different implementation language to blunt recall, but the memory term itself is not measured. On in-corpus repositories, success mixes remembering with reverse engineering in unknown proportions. We therefore report the full cleanroom condition: the model receives documentation and an executable only reference binary, while the source, original tests, and git history remain absent.

The memorization paradox as a training lever

The paradox separates two things that usually blur together: what a model knows and what a model can do with what it knows. Three training and diagnostic uses follow.

1. In-corpus RL could test access to prior knowledge. If training-set membership were independently verified, improvement on exposed repositories could test whether RL helps a model retrieve and coordinate prior representations.
2. Out-of-corpus RL could test transfer. If matched controls verified that a repository was absent from pretraining, improvement would more cleanly support transferable reverse engineering skill rather than recall.
3. Cross-corpus checkpoints could become a diagnostic. With verified membership labels and matched tasks, checkpoint-level score changes could help distinguish retrieval from newly learned problem-solving behavior.