Vetto Science Bench
Each task puts an agent in a terminal with realistic data and a research question. To solve it, the agent has to explore the data, choose a method, check whether that method's assumptions hold, and deliver an answer or a working analysis pipeline, which we grade against held-out ground truth.
Model results
A trial passes only if its grade lands in the correct range and it clears both the trajectory audit and the leak audit, so the board ranks one number: the share of trials that pass. Line 2 is the setting each model ran at — its reasoning effort, and tool mode where each step is a native tool call.
GPT-6 Astra OpenAI · max 49.0% 
Gemini 4 Argon Google DeepMind · high, tool mode 46.6% GPT-6.1 Sol OpenAI · max 28.9% 
Claude Opus 5.5 Anthropic · xhigh 24.2% Claude Sonnet 5.5 Anthropic · xhigh 18.2% Claude Haiku 5.5 Anthropic · max 16.2% Claude Fable 5.1 Anthropic · high 9.4% 
DeepSeek V4.1 Flash DeepSeek · max, tool mode 8.0% GPT-6 Luna OpenAI · max, tool mode 6.7% 
Grok 4.7 xAI · xhigh, tool mode 6.7% Gemini 3.8 Flash Google DeepMind · high, tool mode 6.0% 
GLM 5.3 Z.ai · high, tool mode 6.0% 
Kimi K3 Moonshot AI · high, tool mode 6.0% Claude Haiku 5.5 Anthropic · xhigh 5.3% Pass rate. A trial passes only if its grade lands in the correct range and it clears both the trajectory audit and the leak audit. 50 tasks, three trials per task and model, using the mini-swe-agent scaffold.