Text-to-Video vs Image-to-Video: Pick the Right Starting Point

Every AI video generation starts with a decision that matters more than your model choice, your prompt, or your budget: does the model invent the first frame, or do you hand it one? That is the entire text to video vs image to video question, and it’s the most underrated realism decision in the craft. Starting from a real photograph is the single biggest realism cheat available — and knowing when that cheat stops working is the other half of the skill.

This post deepens Step 1 of our working pipeline for realistic AI video — pick models per shot, not per project. Same principle, one level down: pick your starting point per shot.

What the two modes actually do

Text-to-video is the one in the demos. You describe a shot in words and the model generates every pixel from nothing — subject, lighting, lens, motion, and increasingly the audio too. Total freedom, total invention.

Image-to-video flips the contract. You supply an image — usually it becomes the literal first frame — and the model’s job shrinks to generating motion from that anchor. Some models also accept an end frame, or reference images that steer identity without being frames at all. The variants don’t change the fundamental trade: in one mode the model imagines your subject, in the other it inherits it.

That inheritance is the whole game. A video model has to solve two hard problems at once — what things look like, and how they move. Hand it a real photograph and you’ve solved the first problem with a tool that’s been reliable since the 1800s. The model can spend all of its capacity on the second.

The product-label problem

Here’s the example we’ll carry through the post, because it’s where the difference stops being theoretical.

Say you sell hot sauce, and you want a five-second product shot for an ad: bottle on a kitchen counter, morning light, a little steam from a skillet behind it.

Prompt that as text-to-video and the result will look great — genuinely great, first-glance perfect. Then you look at the bottle. The label reads something like “HOT SAUSE,” in a font you’ve never used, above an ingredients block of confident gibberish, at 12 fl zo. Generated typography still melts under a second look; it’s a standing item on every AI-video detection checklist, and as of mid-2026 no frontier model has fully fixed it. Worse, this isn’t just an artifact — it’s a wrong product. Anyone who owns your sauce knows instantly, and an ad that misrepresents your own label is a problem beyond aesthetics. (We cover label integrity for stills in AI-generated product photos; video inherits every one of those failure modes and adds motion.)

Now do it the other way. Photograph the actual bottle on your actual counter with your phone — two minutes of work. Hand that frame to an image-to-video model and ask for gentle steam drift and a slight push-in. The label is perfect, because it isn’t generated. The lighting is real. The counter is yours. The model cannot misspell a label it was handed.

One caveat, and it previews the whole next section: that guarantee holds only while the label stays roughly where your photo put it. Ask for a slow orbit around the bottle and the model will happily invent the back label — it never saw one. Photographic truth is an anchor, not a force field.

Why a real photo is the biggest realism cheat

Because most of the things that make footage read as real are properties of the first frame, and a photo gives you all of them for free:

• Optics and noise. Real lens character, real sensor grain, real depth of field — the qualities you otherwise spend prompt tokens begging for. This is why prompting like a camera matters so much in text-to-video: you’re describing in words what a photograph simply is.
• Lighting that obeys physics. The shadows in your kitchen photo are correct because photons did the work.
• Identity. It’s your product, your room, your face — not the model’s statistical guess at one.
• Consistency across shots. The same reference stills can anchor every shot in a campaign, which is also the backbone of character consistency.

Framed against the second-look standard: a real first frame means everything static in the shot already survives a second look. You only have to win on motion. That’s a much smaller war.

Where image-to-video loses

If image-to-video were strictly better, this post would be a paragraph. It isn’t, and the failure modes are specific enough to name.

Motion range: the photo’s authority decays fast

The source frame governs the first moments of the clip strongly, and then its influence fades — every frame after the first is generated, and generation compounds. Modest motion (a push-in, drifting steam, a head turn) stays inside the photo’s authority. Ambitious motion — orbits, walk-throughs, anything that reveals geometry the photo didn’t capture — forces the model to invent, and now you’re running text-to-video with extra constraints. A camera move that swings past the bottle asks the model for the one thing you couldn’t photograph in advance: everything else. Plan shots so the camera never asks about parts of the world your photo didn’t show.

Over-animation: nothing is allowed to hold still

Image-to-video models are trained on video, and video moves — so they tend to animate everything. Your locked-off product shot acquires drifting steam you didn’t ask for, a label that gently breathes, reflections that swim, a camera that sways like the whole kitchen is underwater. It’s subtle, it’s everywhere, and it fails the second look precisely because real locked-off footage is genuinely still. Mitigations exist — prompt explicitly for a static camera and minimal motion, generate several takes, keep the calmest one — but as of mid-2026 you’re managing the tendency, not eliminating it.

The frozen-photo look: the opposite failure

Under-animate and you get the tell on the other side: a clip that reads as a photograph with a ripple. Viewers can’t always articulate it, but they feel that the scene was frozen and then thawed. It’s worst with people. Animate a smiling still into speech and you get something best described as haunted portraiture — the expression arrives pre-made and then moves wrong. Humans generally need to be born moving: generated in motion by text-to-video, or driven by a purpose-built avatar system, rather than resurrected from a snapshot.

Dialogue wants to be generated whole

Current frontier text-to-video models generate speech, lip motion, and room audio together, and the coherence shows. Getting dialogue out of an image-to-video pipeline usually means bolting on a separate voice and lip-sync stack — workable, it’s how avatar systems operate, but it adds moving parts and the sync drift that comes with them.

Text to video vs image to video, shot by shot

The running decision, as a table you can apply to a shot list:

Shot type	Start from	Why
Product hero, label visible	Image-to-video, real photo	The model can’t misspell a label it was handed; keep motion modest
Product in use (hands, pouring)	Image-to-video, short beats	Hands and liquids fail hardest; cut before the physics gets ambitious
Talking human, scripted lines	Text-to-video with native audio, or an avatar system	Speech, lips, and room tone cohere when generated together
Recurring character b-roll	Image-to-video from reference stills	Identity lock beats re-describing a face every take
Environment / scenery b-roll	Text-to-video	Viewers have no ground truth to check; cheapest realism available
Big motion or action	Text-to-video	A source frame becomes a constraint the model exits awkwardly
Your actual location	Image-to-video, real photo	Anyone who’s been there is a walking detection checklist

If a shot isn’t on the list, three questions settle it:

1. Is there ground truth in frame? A label, a logo, a face, a room someone knows. If yes, start from a real capture of it.
2. Is the shot defined by motion or speech? If yes, text-to-video — or an avatar system for long scripts.
3. Neither? Use whichever is cheaper on your stack, and spend the attention you saved on the shots above.

The part nobody puts in the demo reel

The uncomfortable implication of everything above: if image-to-video from real photos is the biggest realism lever, then photography is part of your AI pipeline now. The best image-to-video results we see come from people who treat the source photo as seriously as the generation — clean lens, window light, a dozen angles of the product, shot once and reused for months.

That’s unglamorous advice. It’s also an afternoon of work that upgrades every video you make afterward, which is a better return than most prompt tricks. The corollary is worth saying too: a bad source photo is a ceiling. Image-to-video faithfully preserves your harsh flash, your cluttered background, and your fingerprints on the bottle. It’s an amplifier for the frame you give it, in both directions.

The takeaway

Text to video vs image to video isn’t a model debate — it’s a per-shot decision, and it’s mostly about ground truth. If anything in the frame can be checked against reality — a label, a face, a place — start from a photograph of the real thing and ask the model only for motion. If the shot lives on motion or speech, let the model generate it whole. And when image-to-video starts fighting you — over-animating a still scene, or thawing a person badly — that’s the tool telling you that you picked the wrong starting point, not that you need a better prompt.

Concretely, next: storyboard your next video into 5–8 second beats, tag each beat text-to-video or image-to-video using the table above, and spend one afternoon photographing your product properly before you generate anything. If you’d rather have this decision taught with current tools and worked examples — alongside the rest of the pipeline — that’s what Realistic AI Club is: photorealistic AI video as a repeatable craft, ten dollars a month.