Plot Twist Bench

The question every model missed

Three small clusters of objects on a table, filmed from directly above, and a hand that moves a couple of items between them. Twenty-seven seconds, one fixed camera, nothing cut and nothing hidden.

The recording · 27s · 848×478

We asked twelve frontier models one question about it: which cluster reveals an object that was not perceivable at the start?

The answer is the rightmost. Three buttons were placed there and only two could ever be seen, because the third was resting on top of one of the others. When the hand lifted it away, the button underneath became a separate object for the first time.

None of the twelve got it. Nine answered leftmost, the cluster the button arrives in. The other three, all of them Gemini models, answered that nothing was revealed at all.

That question is one of the 48 in the set. To understand why Gemini 3.1 Pro got it wrong, we put two more questions to it once the graded run was over.

The model tracks the moved object correctly, so it names it in the second answer. However, it never checks that fact against the question, and doesn’t notice the gap.

This is the core problem this bench measures. The model can name the objects and describe the scene correctly, but it does not track what actually happened on screen: it settles on a description early and defends it, rather than checking it against what’s on screen. That gap is why it gives a confident wrong answer.

Recognition is not perception

A vision-language model predicts the most likely continuation of what it’s given. Most of the time that looks like watching, because the likeliest description of a scene is usually the true one.

Two abilities blend together here, and this set separates them. Recognition names the objects and the type of scene; it can be answered from what usually happens. Perception tracks what actually happened on screen; it cannot be answered that way.

Telling them apart needs footage built for the purpose, so we filmed our own: thirteen scripts, twenty-six takes, never published online, each built around one twist decided before filming. That is Plot Twist Bench: 48 questions over 26 recordings.

The leaderboard

How the score works. Twelve models, 48 questions. Each model answered every question once per ordering of its options, so a four-option question was asked four times, with the options rotated each time. The headline number is pass@1: the average across those rotations. A question answered right under three of four orderings scores 0.75. Since attempts differ only in option order, that average is the model’s chance of a right answer on a single try. Circular accuracy (CircularEval) is the stricter measure: it counts a question only if the model is right under every ordering. Grading is a plain string comparison against a human annotated golden answer.

pass@1: 75% at the top, people at 85%. 12 models · 48 questions · exported 26 Aug 2026. The strict companion, right under every ordering, ranks the same models on the leaderboard page and in the appendix.

What the scores show. The field spans a wide range, from 75% at the top to 41% at the bottom, and every model falls short of the human baseline of 85%. The two scores diverge for two reasons: missing the answer outright, or getting it right once and failing to repeat it. That second failure is uneven, costing steady models little and unstable ones a lot. Seeing the scene once is within roughly 10 points of the 85% human baseline; holding onto what was seen is where the field comes apart.

Those pass@1 and circular-accuracy numbers carry real uncertainty, and the interval on each row is deliberately conservative: because a model’s answers on the same clip move together (within-clip answer correlation, ICC, is roughly 0.2), we cluster uncertainty by video rather than by question, widening the bars but keeping them honest. How the intervals are computed →

That gap between seeing something once and holding onto it comes down to consistency, which breaks into two shapes, and recent work argues for keeping them apart (Romero-Alvarado et al., 2026). A model can be consistently wrong, repeating the same mistake every reordering, or inconsistently right, slipping between correct and incorrect readings. The repeatability cost itself ranges from about seven points for the steadiest models to as much as thirty-one for the most unstable. Claude Sonnet 4.6 and GPT-5.5 are both 71% consistent, yet Sonnet trails by 25 points on accuracy: one holds firmly to the wrong answer, the other doesn’t waver at all. Consistency per model →

All of that instability sits below the same human ceiling, which is worth grounding in specifics: five testers who had never seen the questions answered them cold and still cleared it, scoring between 84% and 88% and averaging 85%. This is the standard the model gaps are measured against. How the baseline was measured →

What each model actually saw. Some of that score gap traces back to input, not just ability: each row shows a model on its best available video path, not a common setup for all twelve: the three Gemini models took a video file directly, the rest ran on stills with a transcript and spectrogram. What each model received →

Each model ran on its best available setup by design, and the baseline comparison still holds up to a point: three models, GPT-5.6 Sol, GPT-5.5 and Kimi K3, fall within the human baseline’s margin of error on a single pass, but no model matches humans once the same question is asked with reshuffled answer options.

What this extends

DeepMind’s Perception Test (Pătrăucean et al., NeurIPS 2023) is the work that convinced us to film rather than scrape, and it’s the closest predecessor to this one. We took the premise from it; what’s changed since is the score. At publication the best models managed 46.2% against a 91.4% human baseline. By the challenge write-up in January 2026, the top entries had reached 83.7% on the original multiple-choice set.

Those entries were ensembles built on Seed 1.6 Vision, Qwen-VL-Max and Qwen-2.5VL. Every model on our board is a generation newer and stronger. The best of the twelve averages 75% here, and holds 67% when the answer has to survive every reordering of its options.

Price does not buy sight

The figures are metered spend over the same 26 recordings, every model called on every one, normalised to what one clean run of a question costs. The quality axis is the headline pass@1, and the units are defined in the appendix.

pass@1 against the cost of one question · observed spend normalised to one clean run, not published token rates · x axis log scale. The per-model figures are in the appendix, under What each cost figure means.

Three models hold the frontier, and two of them are one model a version apart: Gemini 3.6 Flash at $0.021 a question for 70%, Gemini 3.7 Flash at $0.042 for 72%, and GPT-5.6 Sol at $0.426 for 75%. The other nine sit off it, each matched or beaten on both axes at once, so no budget makes them the right pick on this set.

Sol’s 3-point lead over Gemini 3.7 Flash sits well inside what this set can resolve; its 10× price does not. GPT-5.5 matches Sol’s 75% at a sixth more spend. Kimi K3 sits two points under Sol at 2.5× the price. The sharpest case sits inside one family: Gemini 3.6 Flash is 62% cheaper per question and 14 points better than Gemini 3.1 Pro.

Task design

Each case below opens one recording: the clip, the question as the model received it, the shooting script behind it and every model’s grade. Only three appear here, because this section is a deep dive rather than a measurement: each case is picked to show one failure clearly, not to stand in for the whole set. Two of the three were failed by every model on the board; the third was passed by just one of twelve.

That selection is why the mechanism named in each case is our own reading rather than a measured rate: nothing here says how often the failure happens across the set. For that, the failure landscape has the rates, across all 48 questions and all twelve models.

The failure landscape

Every mode, every model

Every question is tagged by hand, before any model runs, with the failure modes it is meant to expose, and those tags were audited again after the runs are graded. That gives a rate per model per mode: the share of that mode’s questions the model passed, where a pass is the strict one, right under every ordering. All 20 modes this set exercises, ordered by how much evidence each carries.

Share of each mode's questions passed under circular accuracy. Columns, in full: Sol is GPT-5.6 Sol · Kimi is Kimi K3 · GPT5.5 · G3.6F and G3.7F are Gemini 3.6 Flash and 3.7 Flash · Terra is GPT-5.6 Terra · Opus5 is Claude Opus 5 · Qwen is Qwen 3.7 Plus · Grok is Grok 4.3 · Sonnet is Claude Sonnet 4.6 · Op4.8 is Claude Opus 4.8 · G3.1P is Gemini 3.1 Pro. The twelve modes from Multi-stream parallel tracking down carry four questions or fewer: evidence, not a rank.

The table reads the same 240 rates down one column: what the hardening target, Gemini 3.1 Pro, holds on each mode against the other eleven models' average, and the gap between the two.

Eight modes with five or more questions

How the set was built

Every recording starts as a written scenario that names the failure it’s meant to provoke; the scene is then built to trigger it. Each scenario targets one twist aimed at one shortcut, written down before filming starts. Here is one of them, for a scene about making a cup of tea:

The real task (making a cup of tea) is deliberately left INCOMPLETE in an adversarial way: the person pours hot water into the mug but NEVER puts the teabag in. […] Keep the teabag visible on the counter the whole time so the omission is verifiable from the frame, not just inferred. If the teabag ever enters the mug, retake.

A take that breaks its scenario isn’t fixed in editing. It’s reshot. The ground truth is written against the take actually filmed, not the script that asked for it.

On fairness: hard for the right reason

A low score is only interesting if the difficulty sits in perceiving the video, rather than in a confusing prompt, a wrong expected answer, over-demanding grading, or a question answerable from the text alone, any of which would mean the score measures our instrument rather than the model.

Our own pipeline runs into that conflict directly: a question gets harder by iterating against a model until it misses, but that’s also the fastest way to create difficulty that has nothing to do with the video, so failing a model never earns a question its place. Instead, the set is cut from the top of a ranking that never looks at model performance, led by a Q-Score: a 0 to 100 rating of the question purely as a piece of writing, judged from its text alone, for whether it’s unambiguous, stands on its own, and can be graded. A question with two defensible answers scores zero, not something in between.

The check that settles it is the human baseline already on the board: five people with no hand in writing the questions answered them correctly 85% of the time, and a confusing question, a wrong expected answer, or one that can’t be answered from the video wouldn’t have scored that well. So whatever the models are running into here is in the footage, not the instrument. How the baseline was measured →

How a question gets into the set. Seven stages, nine acts. Every act is described in words under its drawing.

What that pipeline cost in human hours

Every question carries the work of at least two people, a median of 2.9, because the person who hardens a question is never the person who audits it. No question in the set is one annotator’s judgment of their own work.

The takeaway

The models on this board beat most humans at mathematics and competition coding. Show them a hand moving buttons around a table for twenty-seven seconds, and the strongest of them leads with 75% pass@1, trailing five ordinary people by ten points.

Three models get close to how people score on average: GPT-5.6 Sol and GPT-5.5 hit 75%, Kimi K3 hits 73%, against the testers’ 85%. That’s close enough that, on just 48 questions, we can’t tell them apart from a single person taking the test once. But not one of them beats even the worst human tester, who still scored 84%. Then just shuffle the order of the answer choices, and the picture falls apart. Every model drops, some by as little as 7 points, one by a full 31. After that shuffle, not a single model comes within 18 points of how people do. The reason is simple: a model settles on a description of the scene early and defends that description rather than checking it against the footage, so one answer only tells you what story it committed to, not whether that story is true.

Until then, ask a video model twice.

Caveats

What these numbers cannot tell you, volunteered rather than waited for.

• A row scores a transport and a model together, not the weights alone. Only the three Gemini rows ran on a first-party video path. For seven of the other nine, stills are what the model ships: GPT, Claude and Grok take no video on any API we could call. The last two are a different case. Kimi K3 and Qwen 3.7 Plus both take video on their own APIs and neither got it here, so read those two rows as image-vision scores.
• Read the ordering with the intervals in view. One question is worth 2.1 points. The interval shown is the conservative one, clustered on the 26 recordings rather than the 48 questions. Rows within about ±12 of each other on the headline metric, or ±17 on the strict one, are too close for this set to rank: the middle of the table reads as a band, not an order. The bottom is separated on both readings.
• Selection is quality-gated, not difficulty-gated. These questions survived a fairness audit; they are not the ones any model fails hardest, so they are not comparable to our difficulty-gated sets.
• The three sample tasks are a deep dive, not a sample. Each was chosen because it shows one failure clearly, so they cannot say how often that failure happens.
• Cost is observed spend normalised to one clean run, not a rate card. Raw request counts differ widely across models after retries and reruns, so each model’s spend is divided by its raw requests and multiplied by the calls one clean run needs. The three Gemini rows are metered from Google’s own billing rather than the gateway.

What’s next

More questions, and more recordings. This is the top priority. At 48 questions over 26 recordings, the interval is wide enough that much of the board reads as a band, not a ranking. The video is the sampling unit, so a new recording narrows that interval faster than another question on a clip we already have. The next round grows the corpus first, and the question count with it.

Fixed-Context beside the native harness. The same corpus already runs a second way: every model, including Gemini, gets identical stills at 1 fps and the same transcript. Two numbers per model on the same 48 questions is the only clean way to see how much of each gap is transport rather than perception. Until it ships, the Gemini-last result can’t be called a perception finding on its own.

More evidence per failure mode, and a wider human panel. Twelve of the 20 modes carry four questions or fewer, too thin to rank. The baseline also rests on just five testers. So the next round widens the panel and gives each mode its own human number, not just the set as a whole. Alongside that, we’ll mark by hand the exact frames each answer lives in, turning “the model missed it” into “the model missed it in these 1.8 seconds”.

Longer videos, and more original scripts. Both push the corpus further from the distribution these models trained on.

Appendix

Everything here is detail the sections above lean on without stopping to explain.

The harness each model got

The aim was to give every model the best video path we could actually reach, rather than flattening all twelve onto a lowest common denominator. In practice that meant meeting each API at its own highest capability.

The three Gemini models were called first-party, through Google’s own Files API, so they got the recording itself as a video file with its native audio track. The other nine were called through Vercel AI Gateway, which is what “stills” in the table below means: the gateway carries images but no vendor video format, so each of those models got still frames cut at its own documented maximum, from 1024px up to 2576px, plus a transcript of the audio and a spectrogram so that nothing in the soundtrack was lost to them.

Kimi K3 and Qwen 3.7 Plus can both take a video file on their own APIs. Neither got that here, and the two rows arrived there differently. Kimi was never sent the file: we cannot reach Moonshot’s video upload through the gateway we use, so from the first request Kimi saw stills we cut from the recording, at 1920px. Qwen was sent the file on all 26 recordings and the gateway refused it all 26 times, so Qwen also saw stills, at 1024px. Either way what arrived was sampled frames, a whisper-1 transcript and a spectrogram rather than video, which is why those two read as image-vision scores.

The other seven stills rows are not fallbacks. GPT, Claude and Grok take no video input on any API we could call, so stills are what those models actually ship.

Resolution and image count follow each provider’s own documented limits. Frame density is the one thing we hold constant rather than maximise, at a target of one frame a second, so that the board compares models rather than frame rates. Stills are sampled uniformly with no keyframe selection, and past a model’s image budget a long clip is tiled into 2×2 grids rather than thinned, so “47 frames” is a mean over emitted images, from 17 to 159, not 47 instants of footage. Grading is identical on every row: temperature 0 except Kimi K3, which its provider fixes at 1.0, so that row’s circular grading is the one that is stochastic.

What each column on the leaderboard means

Accuracy (CircularEval)

The strict companion to the headline. A question is asked once per option ordering and counts only if the model answers every ordering correctly, scored by comparing its answer against the option text. The task-design cards and the failure landscape read on this metric, because a pass there should mean an answer that held.

Q-Score

A 0 to 100 rating of a question as a piece of writing, and the first of the three quality gates. It is judged from the question text alone, never from whether a model got it right, and it asks whether the question is unambiguous, self-contained and gradable. A validity defect, two defensible correct answers being the clearest case, scores zero instead of being averaged away, so a broken question cannot survive on the strength of its other marks.

Chance

Depends on the strategy, which is why we do not quote a single chance rate. A uniform guesser, drawing again in every ordering, almost never comes up correct in all of them. A model that answers by position, always the second option or always the last, scores exactly zero. A model that always names the same option text passes every question whose answer is that option, which is far above either. Read the bottom of the board as evidence about the model, not against a single floor.

Resolution and the 95% interval

One question is worth 2.1 points of a score. The interval is cluster-robust with the video as the sampling unit: several questions on one clip are not independent trials, so they are not counted as such. Where a clip’s questions move together the interval widens, and that clustering runs from about zero to +0.42 by intra-cluster correlation depending on the model. On the strict metric the result sits near ±17 points at 50%, wider than a per-question binomial would give and the honest figure for a set of 48 questions drawn from 26 recordings; the headline’s bootstrap intervals run a few points narrower. Rows within that of each other are close enough that this set does not order them; 25% (11–39%) clears none of the top five. Per-model intervals and ICC →

pass@1

The headline: the chance the model answers a question right in one attempt. Attempts here differ only in option order, so each question is scored as the share of its orderings answered correctly, which is the unbiased estimator of pass@1 under these measurement conditions. The annotator’s original ordering alone is not: that is a single draw, and it can flatter a model by several points (GPT-5.5 answers 83% on the original ordering against 75% averaged over all of them). Because no ordering is systematically harder, the distance from this column down to circular accuracy is the price of inconsistency alone.

Consistency

The share of questions answered identically under every ordering, right or wrong. It separates a model that is stably wrong from one whose answer moves with the options, and it reads as a propensity rather than a capability: how firmly a model commits, not what it can see. On this set it runs from 35% to 79%, rising with accuracy at a rank correlation of 0.71 without being settled by it. The measure is all-or-nothing per question, with four to six orderings each, so it reads harsher than a per-rotation count would: Gemini 3.1 Pro sits last on it at 35%, and its most frequent answer still accounts for 78% of its rotations.

How the 95% interval is computed

Forty-eight questions is not forty-eight independent trials. They come from 26 recordings, and questions asked about the same clip are not independent of each other: if a model misreads the scene, it tends to miss every question on it. Treating them as independent would make the intervals look tighter than the evidence supports, so the video is the sampling unit and the interval is clustered on it.

How much that costs a given row depends on how strongly that model’s outcomes clump by clip, which is what the intra-cluster correlation (ICC) reports. Near zero means a clip tells you little about the next question on it, and per-question counting was roughly right. High and positive means the model passes or fails a recording more or less as a whole, so its effective sample is closer to 26 than to 48 and its interval widens accordingly. A small negative value is not a paradox: it means questions on one clip disagree slightly more than questions drawn at random, so there is no clustering to correct for.

The ICC column and the circular intervals come from a cluster-robust variance with the video as the unit; the pass@1 intervals come from a cluster bootstrap, resampling the 26 recordings twenty thousand times, which respects the same unit without assuming a distribution for the fractional scores. The pass@1 intervals run slightly narrower, about ±12 against ±16 near the middle of the table, because a share of orderings is a less noisy reading per question than all-or-nothing. Read the ICC column as a property of the model on this set rather than of the set itself: the same 26 recordings give +0.42 for one model and −0.04 for another.

Consistency per model

Consistency asks a different question from accuracy: not whether the model was right, but whether it gave the same answer at all. Every question is put once per option ordering, four to six times depending on how many options it carries, so each row below is read over the same 48 questions as the board.

The pass@1 column scores each question as the share of its orderings answered correctly, the unbiased estimate of one attempt: unlike accuracy on the annotator’s original ordering, it favors no particular draw, and the difference is not hypothetical, since GPT-5.5 answers 83% on the original ordering against 75% averaged over all of them. We also checked that the orderings themselves are interchangeable: pooled across the twelve models, the orderings nearly every question carries land within 1.3 points of one another, so no rotation is systematically harder. Everything between a model’s average and its circular score is therefore inconsistency, and that cost runs from 7 points (Kimi K3) to 31 (Gemini 3.1 Pro).

The same answer every ordering column is the strict measure quoted in the text, and it is all-or-nothing: one ordering answered differently and the question does not count, however many agreed. The modal answer share is the same evidence read gently, the average share of a question’s orderings that landed on whichever answer that model gave most often, and the last column says it in plain units. The gentler columns move every model up and change nothing about the order: Gemini 3.1 Pro is last on all three.

Consistency and accuracy rise together at a rank correlation of 0.71, so the rows that break the pattern are where the information is. Claude Opus 5 is steadier than GPT-5.5 and 15 points behind it; Claude Sonnet 4.6 matches GPT-5.5 exactly and is 25 points behind. Both are holding an answer firmly and holding the wrong one, which is a different failure from Gemini 3.1 Pro at the bottom, where the answer itself moves with the option order.

How the human baseline was measured

Five testers worked the set independently, answering the same multiple-choice questions the models were given, from the same recordings, once per question. Nobody was shown a question they had any prior contact with: where a tester had already contributed to one on this project, as its annotator or as its reviewer, that question was withheld from them rather than counted. So each tester saw only footage and questions that were new to them, and the exclusions are the reason two of them answered fewer than 48.

Six question-tester pairs were withheld that way, one for the second tester and five for the fifth, leaving 234 answers of a possible 240. Nothing was skipped: every question a tester was eligible for, they answered. Averaging the five rates gives 85.4% and pooling every answer gives 85.5%, so the figure is 85% either way, and the spread across testers is narrow: 1.4 points of standard deviation, from 84% to 88%.

What it can and cannot be compared with. Each tester saw every question once, so 85% is a single-pass score, and it compares to the headline pass@1, where the strongest models reach 75%. It does not compare to circular accuracy, which requires the same question to be answered correctly under every ordering of its options, something no human here was asked to do. Read against circular accuracy the gap is 18 points, and the difference between those two readings is consistency under re-ordering, which this baseline does not measure. Five testers is also a small panel, enough to establish that the questions are answerable and not enough to place the human number precisely.

What each cost figure means

Spend per request

Metered spend divided by requests billed, per model. The only unit in the export that compares across rows, and the figures below derive from it. It runs $0.005 (Gemini 3.6 Flash) to $0.346 (Claude Opus 5). These are observed prices for this corpus through each provider’s own pipeline, not published token rates.

Cost per question

The horizontal axis of the chart. Raw spend includes retries and reruns, so each task is normalised to one clean run: spend per request × the option orderings that task’s questions need, then summed and divided by the 48 questions. It runs from $0.021 to $1.560, and one clean run of all twelve models costs $291.

Cost per strict pass

Cost per question divided by the share of questions the model holds under every ordering. The figure that charges a cheap model for not seeing: it spreads about 150×, far wider than price alone, from $0.03 (Gemini 3.6 Flash) to $4.99 (Claude Opus 4.8).

Two meters, one wide spread

Nine models are priced from the gateway’s records and the three Gemini rows from Google’s own billing, so Flash against Sol crosses meters while Flash against Gemini 3.1 Pro does not. The raw spend behind these means is far noisier than the normalised figures: retries ran as high as 366 requests where a clean run needs four, which is what the one-clean-run normalisation removes.

The figures behind the chart, per model, cheapest first:

What each failure mode probes