Turn a video into a prompt

get_playbook(task: "video-to-prompt")

Somebody hands you a clip and says: make me this. The job is not to describe the video. A description reads well and produces nothing. The job is to write what gets typed into a generator, which is a different document with different rules.

The difference shows in one sentence. "A moody, cinematic shot of a woman in a kitchen" is a description. "A woman in her early thirties with light olive skin and dark brown hair tied back pours coffee at a pale wood counter, hard morning light from a window on the left, the background well out of focus, fine grain" is a prompt. The second one is longer, and every clause is a thing the model can draw.

Describe. Do not name.

This is the rule that decides whether the output works, and it is broken by instinct, because naming is how people talk about pictures.

Never name a director, a film, a studio or a look that belongs to one. A name resolves inside the model to an average of everything it saw under that name, which is not this video. It is also a claim the frames do not support.

Never name a camera body, a lens in millimetres or a film stock. None of them is visible. What is visible is that the background falls out of focus, that the edges of the frame bend, that the grain is coarse, that the highlights bloom. Those sentences transfer. "Shot on 35mm" does not, because the model has seen that phrase attached to every kind of picture.

Never name the person on screen. Write the age range, the hair, the build, the skin tone and the clothes. A named person is a face the model either refuses or invents.

The test for any line you write: could somebody with no reference draw this? If the line only works for a reader who already knows what you mean, it is a description again.

The pass

1. get_transcript(url). It costs no image tokens and it settles three things at once: whether anybody speaks, in what language, and at what pace. A video with speech needs a different plan from one carried by music.

2. watch_video(url). The whole thing, in one call. The cut times are your shot boundaries. Count them before you write a word. The number of cuts and the length together tell you how many clips this becomes. Keep the session_id.

3. watch_video(session_id=…, question=…) on the shots that carry the look. Not on all of them. The look of a video is usually set in two or three shots and repeated in the rest. Ask about the direction of the light, the colour of the shadows and what the camera does.

4. watch_video(url, window=[t, t+1]) for the texture and for every word on screen. Grain, compression, halation and skin detail exist only at full resolution, and so do letters. A word you half-read is a word you have invented.

Every timestamp you write comes from the stamp burned into the frame. A timestamp you worked out from how far along a sheet looked is worse than none, because somebody will seek to it.

A shot is not a clip

This is the part people get wrong, and it is the part that makes the output usable or useless.

A shot is an observation about the video in front of you: it runs from 4.2 to 8.6 seconds, it is a close-up, the camera pushes in. A clip is an order you place with a generator. They do not match one to one:

• A generator makes 5 to 12 seconds at a time. A shot of 20 seconds is two clips, and the second starts on the last picture of the first.
• A burst of fast cuts inside one setup is one clip. Six quarter-second cuts of the same hands are one order, cut up afterwards in an editor. Ordering six clips gets six different pairs of hands.
• A cut between two places is always a clip boundary. No generator holds a scene change reliably inside one clip.

So write the shot list from the video, then write the clip list from the shot list, and expect the two to have different lengths.

Every clip takes two prompts

Most video generators start from a still image. The still is made first, by an image model, and the video model animates it. That means one clip carries two pieces of text, and they go to two different places:

The first frame. Written the way an image model reads: the subject, the clothes, the place, the framing, the light, the palette, the texture. In full. The image model has read nothing before it and has no idea what the last clip looked like.

The motion. One or two sentences on what happens, plus the camera move. Keep it short. A motion prompt that runs long makes a model average the whole thing into drift, and the movement you asked for disappears.

Then name what the clip ends on. This is the field that turns a set of clips into a sequence: a generator that takes an end image gets it directly, and one that does not takes the last picture of this clip as the first image of the next. Either way, somebody has to know what that picture holds.

The model remembers nothing

A video model reads one clip at a time. Between two clips it retains nothing at all: not the face, not the shirt, not the light.

So every clip repeats the character and the look in full, word for word. "The same woman, now outside" produces a different woman. Write the sentence once, carefully, and then paste that exact sentence into every clip she appears in. Changing the wording between clips changes the person, even when the meaning is the same.

The same holds for the look. If the light comes from the left in clip one and you leave it out of clip two, clip two is lit from wherever the model likes.

When the video is itself generated

Say so, and say what shows it. Then treat it differently, because a fault is not a target.

The usual tells: a hand that gains or loses a finger between frames, a word on a sign that melts into letter shapes, a shadow that falls the wrong way, a reflection that does not match, a surface that slides under a moving object, skin with no pores at full resolution, and background people whose faces reset.

None of these belongs in a prompt. Somebody rebuilding this video wants the look and the motion, not the errors. Write the video the original was trying to be.

Name no model and no product. You cannot see which generator made a clip. The tells above are common to all of them, and a guess dressed as a finding is worse than the silence it replaced.

The useful half of the observation is different: a generated original is reachable. There is no real place, no licensed track and no real face in it, so the usual reasons a rebuild fails do not apply here. Say that too.

Say what will not come back

A prompt that promises the whole video is a prompt somebody trusts. The honest list is short and almost always contains the same entries:

• Text on screen. Generators still set letters badly. Quote the words exactly, with the second they are legible at, and say to add them in an editor.
• A real place, a real product, a real logo. These come back as something adjacent. Say what to substitute.
• A licensed track. Describe the tempo, the instruments and the mood. Never name the song.
• A face that has to hold for 30 seconds. Across four clips it will drift. Say to generate several takes and pick the ones that match.
• Speech with exact words. If the words matter, record or synthesise them and lay them over the clips rather than asking the video model.
• Steam, smoke, water and hair. These smear. Generate the shot without them where you can.

Do not

• Write one prompt for the whole video. A single block of 6,000 characters is not an order any generator takes. It is the summary again.
• Refer backwards. "As before", "the same room", "she continues" all mean nothing to a model that reads one clip.
• Deliver vibes. "Cinematic", "premium", "aesthetic" set nothing. Every one of them is replaced by a fact about the light, the lens or the grade.
• Quote a word you did not inspect. Read it at full resolution first.
• Guess the aspect ratio from the sheet. Read it from the frame.

Playhead runs this whole pass as a product. run_playtool(tool="prompt", url=…) returns the look, the cast, the shot list and the clip plan as one structured result, and it is the faster path when the answer is wanted rather than the working.