SlopCode Vetted
Each task is one project built through a sequence of feature requests. Every checkpoint is a new agent session that inherits the repository but not the conversation, and passes only when the hidden suite — its new tests plus every earlier one — passes in full.
Model results
Three readings of a run’s final workspace, one ranking: pick the metric and the board re-ranks by it. Resolved is the share of runs whose last checkpoint passes every test, almost the share ending with at least 95% of tests passing, and mean the average share of tests passing — so a model can pass nearly every test and still finish few projects whole.
GPT 6.1 Sol OpenAI · 90.1% checkpoint rate 83.7% 
Claude Opus 5.5 Anthropic · 86.2% checkpoint rate 75.7% 
Kimi K3 Moonshot AI · 72.5% checkpoint rate 60.4% 
Gemini 3.8 Flash Google DeepMind · 73.7% checkpoint rate 56.2% 
Muse Spark 1.3 Meta · 53.8% checkpoint rate 34.0% 
Grok 4.7 xAI · 47.6% checkpoint rate 23.6% Resolved is the share of runs whose last checkpoint passes every hidden test. 50 tasks, three runs per task and model, one minimal agent scaffold. Line 2 is the mean checkpoint rate: the share of checkpoints whose cumulative suite passes in full.