Free cookie consent management tool by TermsFeed Generator
All Research

Introducing CASTBench

Can video models bring a cast of animated characters to life?

Shweta Bhardwaj
Ankit Khedia
Rachana Yalla
Navaneeth Bodla
Aangeeras Narayanan
Shakeel Imdad

Can video models bring a cast of animated characters to life?

We graded 1,902 clips from 6 reference-to-video models against a fixed human rubric to see how often they turn a multi-character story into a clip that gets everything right.

Say you give a video model a reference image of a fox, one of a rabbit and one of a red umbrella, plus a few lines of story: the fox does a backflip, lands beside the rabbit and hands it the umbrella. The clip that comes back can look polished and still get the story wrong. The fox may match its reference perfectly and never actually flip, or the rabbit may do the flip instead. The umbrella may turn blue halfway through or end up in the wrong paws, and during the handover the two may never touch, or a paw may pass straight through the handle. A cartoon fox is allowed to squash and stretch mid-flip; it is not allowed to lose track of who did what to whom. CASTBench grades clips on exactly these points, one story and one clip at a time, so a result shows which model is ahead and also what each model can be relied on to do.

A story in CASTBench gives the model one written prompt and a reference image for every character and prop, and asks for one short clip in which several of those characters act and touch. Trained annotators then grade every clip on the 13 overall rubrics, and on the capability mode its story was written to test.

Seedance 2.5 leads. Even so, it gets 3.1 questions not fully right in an average clip and produces a fully clean clip, with every question and every scripted action a clean Yes, only 9.5% of the time. For Grok Imagine those figures are 7.2 and 0.6%.

Who leads, and how far every model is from a clean clip

We report the Rubric Pass Rate (RPR). On each clip, every rubric that applies to it (a capability mode applies only to the stories written to test it) scores one if every question and every item in it, such as each scripted action or each listed detail, is answered Yes; one half if the worst answer is Partial; and zero if any answer is No. A model's RPR is the mean of these scores over its clips for each rubric, then the unweighted mean over the rubrics.

The rule is strict on purpose: a clip in which one character's hand passes through another's arm, or in which a scripted hand-off never happens, is not a usable clip, however many other questions it passes.

Leaderboard of the six models: held-out and public rubric pass rate, things wrong per clip and share of fully clean clips
Ranked by held-out RPR. Things wrong per clip counts the questions a clip does not get fully right; a fully clean clip has every question and every scripted action a clean Yes. Both are over all stories.

On RPR over all 317 stories the field spreads from 60% to 82%. Seedance 2.5 leads clearly: it comes first in every one of thousands of re-draws of the stories. Two pairs sit too close to call, Wan 2.7 vs Kling O3 and MiniMax-H3 vs HappyHorse 1.1, so their order is not a finding.

Price separates the models too. We accessed all six on 18–19 August 2026. At their list prices as of 9 October 2026, one 10-second clip from Seedance 2.5 costs $4.73, about 7.9 times the $0.60 of MiniMax-H3, and one clip for every story costs $1,499 against $190. The best value comes from MiniMax-H3, HappyHorse 1.1 and Seedance 2.5: no other model is both cheaper and better than these. Kling O3, by comparison, costs $4.20 a clip for a held-out RPR of 64.1%.

Scatter plot of each model's list price per 10-second clip against its held-out rubric pass rate, with the best-value models joined by a line
Quality against cost. Each logo is one model: the list price of one 10-second clip (across, on a log scale) against its held-out RPR (up). The line joins the best-value models.

The radar below uses a second measure, with partial credit: the Item Pass Rate (IPR) of each rubric. Each question on a clip scores the mean of its answers over the items it asks about, with Yes as one, Partial as one half and No as zero. Each question is averaged over the model's clips, and a rubric's IPR is the unweighted mean over its questions. One wrong answer lowers the IPR but does not zero it.

Two radar charts of each model's item pass rate on the capability modes and on the overall rubrics
The capability modes and the overall rubrics: each spoke is the model's IPR on that rubric (partial credit: a Partial answer counts half). OR10 Fine details is drawn as two spokes, its listed and its unlisted details, because the two are scored differently. Each capability mode is scored only on the stories written to test it; each overall rubric on every clip. The centre is one half and the rim is everything right.

What CASTBench measures

CASTBench uses its 8 capability modes to design the stories, with each story written to test one of them, and its 13 overall rubrics to grade every clip. Together they measure six broad families of capability, in the order a story is read:

  • Who is on screen: the right characters and props appear, each exactly once, and stay distinguishable from look-alikes and twins, including after being hidden (M4 Occlusion, M5 Look-alike, M6 Twin, OR1 Cast & props present, OR4 Cast count).
  • Looking like the reference: characters and props keep the appearance on their reference sheets, down to small details (M2 Prop binding, OR3 Appearance consistency, OR10 Fine details, OR14 Prop appearance stability).
  • Doing the scripted action: hard motions are performed and every scripted action happens, clearly framed (M1 Hard-motion integrity, OR9 Action coverage, OR13 Action staging).
  • Characters acting together: scripted contact between characters happens, in the right direction, and looks physically real (M3 Interaction, M7 Handover).
  • Physical plausibility: bodies and objects sit, rest, move and deform the way the physical world allows (OR2 Motion smoothness, OR6 Placement & spatial sense, OR7 Within-shot stability, OR11 Physics & deformation).
  • Story continuity: the story holds together across cuts and over time, with sensible pacing and props that see their arc through (M8 End-to-end Persistence, OR8 Cross-cut continuity, OR12 Pacing).
Share of clips where each capability mode or overall rubric is not fully right, for all models together and for each model
Share of clips where each capability mode or overall rubric is not fully right. The grey bar is all models together; each dot is one model.

Four parts of the rubric go wrong on more than half of all clips: Action coverage (OR9) on 82%; Interaction (M3) on 80%; Placement & spatial sense (OR6) on 56%; Hard-motion integrity (M1) on 55%. The least failed, Cast & props present (OR1), still goes wrong on 6%. Seedance 2.5 is the best model, alone or level with others, on 19 of the 21. The exceptions are Interaction (M3), led by HappyHorse 1.1, and Cross-cut continuity (OR8), led by MiniMax-H3.

Where each model falls short

Seedance 2.5 is the strongest model overall, but it is not the strongest everywhere, and each of the other models fails in a recognisably different place.

HappyHorse 1.1 and Seedance 2.5 make scripted contact happen most often (72% and 71% of scripted contacts), and these stories cannot tell the two apart. Yet HappyHorse 1.1 lets something pass through something solid more often than any other model (clips free of it: 56%, against 76% for Seedance 2.5). MiniMax-H3 is third on whether contact happens (63%) but, in the handover stories, scores highest for contact that looks physically real (91%) and is clearly staged (96%). Wan 2.7 and Kling O3 keep every character on screen and recognisable almost perfectly while doing far less of what the story asks: Wan 2.7 completes 55% of hard motions and Kling O3 41% of scripted contacts.

Each question in the chart below sits in its rubric's family, with four exceptions. For these, the rubric belongs to one family but what the question asks belongs to another, so we place the question by what it asks. That keeps each family about one kind of failure:

  • M1.2 Identity through motion, from M1 Hard-motion integrity (Doing the scripted action), asks whether a character stays recognisable through a hard motion, so it sits under Who is on screen.
  • M2.4 Prop use sensible, from M2 Prop binding (Looking like the reference), asks whether a prop is used in a sensible way, so it sits under Doing the scripted action.
  • M1.3 Natural deformation, from M1 Hard-motion integrity (Doing the scripted action), asks whether bodies deform naturally during a hard motion, so it sits under Physical plausibility.
  • OR14 Prop appearance: nothing comes loose, from OR14 Prop appearance stability (Looking like the reference), asks whether a held prop comes loose and floats free, so it sits under Physical plausibility.
Share of clips each model gets fully right on every who is on screen question, on a 40% to 100% scale
Share of clips each model gets fully right on every looking like the reference question, on a 40% to 100% scale
Share of clips each model gets fully right on every doing the scripted action question, on a 40% to 100% scale
Share of clips each model gets fully right on every characters acting together question, on a 40% to 100% scale
Share of clips each model gets fully right on every physical plausibility question, on a 40% to 100% scale
Share of clips each model gets fully right on every story continuity question, on a 40% to 100% scale

Each row is a question on which the models differ: the best and the worst model are at least 10 percentage points apart, and the gap holds up across stories. Questions where all six models are close, or the gap is too uncertain to call, are left out. Rows are grouped by the six families above, in the order a story is read: who is on screen, how they look, what they do, how they act together, whether it is physically plausible, and whether the story holds together. Every model sits on the same scale, low on the left and high on the right, and models with close values sit side by side. Use the arrows to move between families.

Two kinds of story are comparatively easy in clips this short. Questions about a character who is hidden for a moment fail 7% of the time, and questions about telling look-alikes and twins apart 7%, while 40% of scripted contacts never happen at all. We did not vary how long a character stays hidden or how alike two characters look, so we do not claim either is solved in general. Some of the gaps between neighbouring models are small and should be read as a pattern rather than a ranking.

Models keep who is on screen yet they lose what the characters do

A character almost always stays recognisable through a hard motion, but the motion itself, and the story's actions, are where clips go wrong.

In the stories built around a demanding motion, the character stays recognisable through it in 97% of cases, while the motion is performed in only 71%. The chart below follows each model's clips on the interaction stories through a series of checks, keeping only the clips that pass every check so far: 69% to 97% of clips have the right characters on screen, and only 5% to 28% still pass once every scripted action must also happen.

Line chart following each model's interaction-story clips through successive checks, from cast present to every scripted action happening
Share of each model's clips on the interaction stories that pass every check up to and including the one named.

Stories get harder as they ask for more. Averaged over models, a story with 1 reference image has an RPR of 90% and one with 4 has 64%; by number of characters the drop is from 91% to 63%. Longer prompts and more scripted actions show the same direction. These factors rise together across stories, so we do not separate them.

Rubric pass rate per story for each model, grouped by reference images, cast size, scripted key actions and prompt length
RPR per story (mean of the 6 models), grouped by how much the story asks for; n is the number of stories in each group.

Characters often fail to act together

When two characters are supposed to touch, the usual failure is that the contact never happens. When it does happen, it is usually a real touch.

In the 39 interaction stories, 40% of the 840 scripted contacts between characters never happen, and only 21% of clips show every one. The cast is not the obstacle: in 87% of these clips every character is on screen and counted correctly. Contact is more likely in those clips (on average 60% of a clip's scripted contacts happen, against 33% when the cast is wrong), but still falls well short.

For each model, scripted contacts that happened, partly happened or never happened, and how often contacts that happened were contact-rich
Left: for each model, its scripted contacts in the interaction stories: happened, partly happened, or never happened. Right: how clearly the contact is a real touch rather than characters standing near each other, in the clips where at least one contact happened in full, with a partly right answer counting half.

The models differ far more on whether contact happens than on how it looks once it does. The share of scripted contacts that never happen runs from 24% for HappyHorse 1.1 to 56% for Kling O3; the real-touch score stays between 82% and 92% for every model.

Props appear, then fall apart

Getting the right prop into the scene is easy; keeping it the same object, in the right hands, through the story is not.

In the stories with props, 92% of clips have every character and prop present, correct and in the right hands. Requiring the prop to keep its look leaves 64%. Requiring as well that nothing worn, held or in the scene pops or vanishes leaves 41%. In the stories that follow a prop's whole life, from handed over to used to its final fate, 20% get every stage right.

Line chart following each model's clips in the prop stories through successive prop checks to a complete prop lifecycle
Share of each model's clips in the stories with props that pass each stage and every stage before it.

Failures stack

Most clips go wrong in more than one way at once.

Of all clips, 91% show at least one of 6 common kinds of failure and 68% show two or more. The most common pattern is action missing on its own (19% of clips), followed by action missing + object pops or vanishes (8%).

For each model, the share of clips showing no failure, one kind, two kinds, three kinds, or four or more kinds of failure
How many kinds of failure each clip shows; the darker the segment, the more things went wrong at once.

What this benchmark produces

CASTBench is released as 50 public stories, each with its prompt and the scripted actions and contacts it asks for, and the human grades of all 6 models' clips: 300 clips and 7,902 graded answers, with annotator identities removed. The other 267 stories are held out, so a new model can be scored on stories no one has tuned to. The scoring code reproduces every number in this post from the grades.

Because every score breaks down into what was asked of the clip, a result can be read at the level that matters for a fix: whether a model fails to start a contact or starts it and breaks the physics; whether a prop never appears or appears and drifts. A single overall score hides these differences.

What this does not show

This is not a general ranking of video models. Each story is one prompt and one short clip per model, so the results describe short, multi-character animation from reference sheets. RPR is deliberately strict: it rewards clips that get everything right, not clips that get most things right. Two pairs of models cannot be separated on it; the held-out stories are the reference.

Some comparisons are made on subsets: contact findings on the stories that script contact, prop findings on stories with props. Prices are list prices, not what any run was billed. How long a character stays hidden, how alike two characters look, and when in a clip a failure starts were not annotated, so we make no claim about them.

1
Models Evaluated 
Seedream 4.0, Higgsfield Soul, GPT Image 1, Flux.1 Kontext and Nano Banana Pro.
2
Task Diversity
Multi-complexity tasks across image generation (novel/knowledge-based) and image editing (with/without preservation) domain.
3
Prompt Design
40 novel prompts constructed from a 5-dimension taxonomy (Use Case × Content Type × Style × Conversation Type × Composition) ensuring diverse coverage.
4
Rubric Dimensions
  • Visual Aesthetics (Simplicity, Diversity, Colorfulness, Craftsmanship) 
  • Quality Adherence (Object & Layout Fidelity, Attribute Fidelity, Edit Precision, Context Preservation, Seamlessness, Text Legibility, Knowledge Grounding) 
  • Creativity & Novelty
  • Fairness & Representation
5
Scoring & Rating
Scored model performance on 4-point Likert scale (1=lowest, 4=highest) and implemented win-rate matrix over N=40 head-to-head comparisons.
6
Quality Control
  • Gold samples injected mid-evaluations for drift monitoring (3 independent annotations per task). 
  • QC adjudication layer for disagreements (~12% of datapoints). 
Note:
  • This is NOT a definitive ranking. It's a structured snapshot under specific rubrics and prompts.
  • Scores do NOT predict performance on prompts outside the taxonomy's covered combinations.
  • Results do NOT account for inference cost, latency, API availability, or pricing — only output quality.

This doesn’t have to end here

Accuracy is Intelligence