Introducing CaptionBench: A Stalemate on the Leaderboard Masks Distinct Failure Modes

A caption used to be metadata. It used to be alternate text, or a keyword for search, and if it said "someone picks up a cup" without getting the exact moment right, nobody noticed or cared. That's no longer true. Captions now drive retrieval, clipping and search directly on caption corpora across video libraries, where the caption is the primary artefact rather than a label attached to one. A caption that reads well but places an action three seconds off from when it actually happens is a real defect there, not a style problem.

That leaves anyone buying a captioner needing two answers: which model fits their footage, and when a model gets something wrong, what fixes it. The tools built to answer the first question are wearing thin, and existing evaluations rarely answer the second.

This piece reports our internal benchmark for video captioning: 196 short-form clips, each captioned by six frontier models. Every model returns two things per clip  a single overview paragraph describing the whole video, and a timeline that splits it into three to five time-bounded windows, each with its segment level captions. Trained annotators judged both against the footage, separately.

Why the usual caption-scoring metrics can't tell a good caption from a bad one

Traditional caption metrics measure similarity, not correctness. BLEU and CIDEr compare generated captions against a reference sentence, which works for short descriptions but breaks down for long video summaries where many different phrasings can be equally valid. CLIP-based metrics improve visual alignment but can still miss temporal mistakes, such as describing an action in the wrong order or confusing camera motion with object motion. Hallucination metrics (CHAIR, POPE) focus mainly on whether mentioned objects exist, a narrower failure mode than the temporal and grounding errors we study here.

We tested whether standard metrics could distinguish captions humans accepted from those they rejected. BLEU-4 and CIDEr were essentially unable to separate the two groups: BLEU-4 achieved an AUC of 0.51 and CIDEr 0.53, no better than chance. Given the best references available from reviewers' own corrections,  BLEU-4 fell to 0.48, scoring rejected captions slightly higher than accepted ones. This was not a length artifact; accepted and rejected captions differed by only one word in median length.

The same limitation appears at the model level. Using metric scores alone, Qwen3.8-27B ranks third of six on BLEU-4. Human reviewers put it last, at 54.1%. A single similarity score can therefore produce the wrong conclusion about which model performs better.

Standard metrics give us a score, but they do not tell us what went wrong or how to fix it. We wanted to understand the errors behind the numbers, so instead of only asking reviewers to accept or reject captions, we asked them to correct them. A correction shows both the mistake and what a better caption should look like.

Why we had reviewers rewrite every caption instead of scoring it

We grouped every correction into one of six failure categories defined before annotation began. Each caption was reviewed twice: as a complete summary and segment by segment summary against the actual footage. This produced 1,144 reviewed captions across 196 clips and six models, with strong agreement between reviewers (κ = 0.947).

This second view exposed failures that are easy to miss in a finished caption. Camera errors were the clearest example: a camera movement described as an object moving, or a pan across a scene interpreted as movement within the scene. Reviewers found these errors 4.7× more often when they checked captions window by window rather than reading the full summary.

Two models are at a stalemate on the leaderboard. Yet, they have completely different failure modes.

Of the six models in this study (Qwen3.8-Max, Muse Spark 1.2, Kimi K3, Seed 2.1 Turbo, Gemini 3.1 Pro and Qwen3.8-27B), Qwen3.8-27B and Gemini 3.1 Pro are the clearest example of the finding the rest of this piece builds on, because they're the pair whose scores come closest to tying.

They land 4.6 points apart on correct-caption rate, 54.1% against 58.7%, on the 172 of 196 clips all six models captioned. 

That gap is too small to call statistically real, so if you stopped at the score, you'd conclude the two models have roughly the same problem and need roughly the same fix.

But that's not the case. 

Qwen 3.8-27B over-asserts. 46.4% of its captions describe something the footage doesn't actually show, an object that isn't there, an action that didn't happen, the worst rate of invention in the study. But it gets the timeline right better than any other model tested; it rarely misplaces what happened when.

Gemini is almost at the opposite end of spectrum showing fewer unsupported factual claims in this benchmark, but it omits details and gets their timing wrong, the worst in the study on both counts, and its captions run shortest overall, as if it's saying less to stay safe.

Qwen 3.8-27B suggests grounding data as the most targeted intervention: examples that pair each false claim it made with what the footage actually shows, so it learns to hold back instead of guessing. 

Gemini needs recall and localisation data instead: complete, correctly-timed example captions, so it learns what "complete" and "correctly placed" actually look like. Fixing one means using data built around correcting false claims. Fixing the other means using data built around filling gaps and timing.

Accuracy and severity rank the models differently

Accuracy, or the correct-caption rate, asks one question: how often did a reviewer accept a caption exactly as the model wrote it, no edits at all? 

The chart above shows the full spread, Qwen3.8-Max highest at 72.7%, Qwen3.8-27B lowest at 54.1%. 

What it doesn't make obvious is that the top four, Qwen3.8-Max, Muse Spark 1.2, Kimi K3 and Seed 2.1 Turbo, can't actually be ranked against each other — the leader is inside the tie, not above it

Their scores and their confidence intervals overlap enough that only 4 of the 15 possible pairwise comparisons across all six models hold up once you apply the Bonferroni correction for testing that many pairs at once. 

In practice: if you picked one of those four over another because it sat one spot higher on this leaderboard, the data doesn't back that choice up.

But accuracy only tells you how often a model is wrong, not how badly. 

Some wrong captions need one word swapped. Others need to be thrown out and rewritten almost from scratch. 

We measured that separately: a caption counts toward the "critical" rate if the reviewer kept less than 55% of the model's original wording while fixing it. It's a rough proxy, since it tracks how much text moved rather than how much the meaning changed, so treat every rate here as a floor, not a precise count.

This is where Muse Spark 1.2 stops looking good. It's second-best on accuracy at 72.1%, which would put it near the top of a shortlist built on that number alone. 

But it's dead last on severity, at 16% critical failures, 2.6 times Qwen3.8-Max's rate. 

And its errors don't spread evenly between mild and severe; they cluster at the extremes, fewer moderate problems than any other model here, more total rewrites. 

When Muse Spark gets a caption right, it's usually clean. When it gets one wrong, it's usually wrong enough to redo from scratch. Rank on the first chart and you'd recommend it. Rank on the second and you wouldn't.

The failure most caption reviews miss: camera movement mistaken for something moving in the scene

How closely a reviewer checks a caption changes how many camera errors get caught, independent of what the model actually did. We had each reviewer grade the same caption twice, once reading the whole paragraph as a summary, and once going window by window, checking each few-second segment against the footage, both times against the same six error categories.

Every category caught more errors on the window-by-window pass than on the summary pass, which isn't surprising on its own, a careful read is bound to unearth more errors than a quick pass. 

Four of the five modes gained by 1.5 to 2.8 times, a fairly consistent bump. 

Camera and shot broke that pattern: it gained 4.7 times, nearly double the next-largest jump. And when reviewers do catch a camera error, it tends to be severe: 21.6% of camera fixes require a near-total rewrite of that segment, against 5.5% for omissions.

What makes the camera different is that segment review forces an explicit call on camera behaviour in every window, where a whole-paragraph summary can describe an entire clip and never once address camera movement directly. There's nothing to catch it against.

The practical upshot is that anyone grading captions by reading the summary alone, whether that's a human reviewer or an automated judge, is structurally blind to the error that's most expensive to fix. And this isn't a weakness in one or two models. 

Pooled across all six, camera error rates sit in a tight band, 10.9% to 14.9%, the narrowest spread of any failure mode. No model in this study consistently avoided camera/shot failures, suggesting that this remains an important gap.

An accepted caption is not a clean caption

The same blind spot shows up at the caption level, not just the failure-type level. A reviewer accepting a whole-clip overview is judging the summary, not certifying every segment underneath it.

Muse Spark hides a rejected segment inside one out of every 2.4 captions a reviewer waved through at the top level. Even Qwen3.8-27B, the cleanest model on this measure, buries one in 28.7% of the captions it got waved through.

Whole-caption acceptance doesn't mean the timeline underneath it is clean. This is the second measure, after accuracy-versus-severity, on which Muse Spark looks fine at the top level and doesn't hold up underneath. The camera example is one failure mode. Across the benchmark, we grouped every rejected caption into one of six dominant error categories.

Failure modes

Every failure in this study is typed against the six modes in the table below. The modes are not exclusive, so each rate says how often that mode appears in a caption rather than which mode dominated it. The rates pool all six models.

Segmentation is judged once per caption rather than per repair, so it gets no severity split and no cost figure. All of these are floors, since modes were only ever recorded on captions that had already been rejected, which means a mode sitting inside an accepted caption never got counted at all.

What the corrections reveal

A caption fails for one of two opposite reasons: the model said too little, or it said something wrong. Those call for different fixes, and we can tell them apart by watching what the reviewer actually did to repair each broken segment. Every repair falls into one of three moves: add something the model left out, cut a claim that shouldn't have been there, or swap a wrong claim for the right one.

We call a model additive when most of its repairs made the caption longer, meaning the reviewer was mostly filling gaps. We call it substitutive when most repairs didn't add length, meaning the reviewer was mostly replacing wrong statements with correct ones. 

That distinction decides how you read the correction. When a model is additive, the corrected caption stands on its own as a complete description of that clip. When a model is substitutive, the correction alone says little; the informative object is the pair, the wrong claim set beside the right one.

Here's what one of these repairs actually looks like on the page, from a Gemini 3.1 Pro caption on a 10.6-second domestic clip, rejected by both reviewers:

Two hands became two people, and the model built an entire interaction between them that never happened. That's the kind of near-total rewrite the critical rate is counting; pooled rates matter more than any single example. 

This is the answer to the question the opening example raised. Qwen3.8-27B is substitutive, 42% of its repairs lengthen the text, 19% shorten it, which fits a model that over-asserts rather than leaves things out. A targeted intervention would likely be grounding data: examples pairing unsupported claims with what the footage actually shows. Gemini is additive, 58% of its repairs lengthen the text, which fits a model that omits and mislocalises rather than asserts wrong things. Gemini's profile suggests a need for more complete, correctly timed examples. Muse Spark 1.2 and Kimi K3 share Qwen3.8-27B's problem, over-asserting, and need the same kind of correction pairs. Seed 2.1 Turbo shares Gemini's problem, omitting and mislocalising, and would likely benefit from the same kind of complete examples.

There's a second, separate distinction worth knowing when you're deciding what footage to go collect: some models only break down under specific conditions, and others break down everywhere regardless of what you show them. Qwen3.8-27B loses 26.1 points of correct-caption rate specifically on handheld footage compared with footage shot on a locked-down tripod. That's the easier problem: you know exactly what kind of footage to go find more of. Gemini, Seed 2.1 Turbo and Qwen3.8-Max don't show that pattern; they're weak fairly evenly across every kind of footage we tested, which is the harder problem, because there's no specific gap to target.

One more thing worth checking, since this whole piece rests on these corrections meaning something real: are reviewers just tightening prose, or actually fixing errors? We checked rather than assumed it. Diffing all 392 rejected captions against their corrections and classifying what each edit changed, 98.7% alter something a reader would come away believing differently, a count, an object, an action, and over half specifically fix a spatial relation, the single most common thing these models get wrong. Human correction is not polishing. It is semantic repair. This is why average caption quality scores are not enough: they tell you whether a model failed, but not what kind of failure happened.

Camera and shot understanding remains a shared weakness in this benchmark: it is expensive when wrong and difficult to detect from caption-level evaluation alone.

Which scenes break models

Every evaluation clip has two different qualities: how difficult it is, and how well it reveals differences between models. Those are not always the same thing. The hardest footage is not necessarily the most useful benchmark footage but what matters is finding examples where models fail in different ways and the differences become measurable.

In our analysis, two characteristics of a clip were consistently linked to whether a caption survived review, and both can be controlled when building an evaluation set.

Kitchens, bathrooms and living rooms are cluttered and close-range and full of hands doing small things to small objects, which is about the worst combination you can hand a describer. Clutter on its own carries a good deal of that: rating each clip for scene complexity independently of what it contains, simple scenes score 69% against 61.3% for moderately busy ones. So it is the density of the scene doing the damage rather than the room it happens in.

Three other dimensions were less predictive than we expected. Indoor versus outdoor footage barely separated performance (3.0 points, p = 0.30). Shot scale was more suggestive but still below our significance threshold (χ² = 10.6, df = 4, p = 0.032): performance ranged from 80% on extreme-wide shots to 60.4% on clips mixing multiple scales.

Motion was the biggest surprise. We expected high-motion clips to be harder, but the effect was not significant (χ² = 3.2, df = 2, p = 0.20). In fact, high-motion clips scored slightly better than medium-motion clips (69.9% vs. 62.1%).

We would not conclude that fast footage is easier but motion alone does not appear to be a reliable difficulty signal. For building challenging evaluation sets, the stronger levers appear to be scene clutter and camera movement, not simply how much action is happening.

Only two clips in the corpus defeated all six models. Most failures were therefore not concentrated on universally impossible footage; they reflected differences in how individual models handled the same videos.

What this benchmark produces

We set out to rank seven captioners and ended up learning more from the gaps between them than from the ranking itself.

Every evaluation produced more than a score: it produced a human correction, a failure label, and a record of what the model should have done differently. The result is a benchmark with 1,342 reviewed captions, 454 rejected outputs with corrections, six consistent failure categories and 5,431 window-level judgments showing where errors occur in the timeline.

These corrections reveal different things depending on the failure pattern. When a model mainly leaves information out, the corrected caption provides a complete example of the behavior we want it to learn. When a model confidently describes something that did not happen, the useful signal is the contrast between the incorrect claim and the corrected version.

Because every error is tied to a failure category, future evaluations can move beyond a single overall score and ask more specific questions: Did the model improve at camera understanding? Is it missing fewer important events? Is its temporal alignment getting better? A single caption score hides these differences.

We built this benchmark to understand where video captioning systems still fail. The methodology is model-agnostic: the same framework can be applied to any future captioning system. The goal is not just to decide which model wins, but to understand what still needs to improve before a caption can be trusted.

What this doesn't show

This is not a general ranking of video models, a claim that any model cannot understand video, or a measure of time or cost saved. The evaluation covers 196 short-form clips, mostly under thirty seconds, with one prompt. What we believe generalizes is the method: similar scores can hide very different failure patterns.

Two caveats matter. Severity is estimated from how much text changed during correction rather than a separate human severity rating, so those numbers should be treated as lower-bound estimates. And reviewers didn't all see the same mix of clips, so some of the small differences between models reflect who reviewed what and we adjust for that statistically and the large gaps hold up, but the models which are a point or two apart in the middle shouldn't be read as ranked.