parsing-video

Parsing Video

Claude cannot play video, but it can read images. To interpret a video, sample frames evenly across its duration, tile them into timestamped contact sheets, and Read the sheets. A 4×4 sheet compresses ~16 moments into one image, preserving narrative flow — what changed, in what order, roughly when.

Requires ffmpeg/ffprobe (apt-get update && apt-get install -y ffmpeg if missing).

Workflow

1. Probe first

ffprobe -v quiet -print_format json -show_format -show_streams input.mp4

Note duration, resolution, and whether there's an audio stream. Duration drives how many sheets you need.

2. Generate contact sheet(s)

python3 scripts/contact_sheet.py input.mp4                          # 1 sheet, 4x4, whole video
python3 scripts/contact_sheet.py input.mp4 --sheets 3               # 48 frames across 3 sheets
python3 scripts/contact_sheet.py input.mp4 --start 120 --end 300    # zoom into 2:00–5:00
python3 scripts/contact_sheet.py input.mp4 --grid 3x3 --tile-width 500  # fewer, larger tiles

The script probes duration, samples frames at interval midpoints, stamps each tile with its source timestamp (H:MM:SS, bottom-left), and tiles them into <name>_sheet_NN.png. It prints each sheet's time range.

Sheet budget — more sheets = more Read calls; scale to duration and task:

Duration	Sheets	Sampling interval
< 2 min	1 (4×4)	~4–7 s
2–10 min	2–4	~10–40 s
10–60 min	4–8, or coarse-then-zoom	~1–2 min
> 1 hour	coarse pass, then zoom	varies

3. Read and interpret

Read each sheet image. Tiles run left-to-right, top-to-bottom in time order; use the stamped timestamps to anchor observations ("the scene changes around 1:42"). Cross-sheet continuity: the last tile of sheet N immediately precedes the first tile of sheet N+1.

4. Zoom when needed

Contact sheets trade resolution for coverage. When something needs a closer look:

# Re-sheet a narrower window at higher tile resolution
python3 scripts/contact_sheet.py input.mp4 --start 95 --end 125 --grid 3x3 --tile-width 500

# Or extract a single full-resolution frame at the moment of interest
ffmpeg -ss 00:01:42 -i input.mp4 -frames:v 1 detail.png

Content-aware sampling

Uniform sampling guarantees temporal coverage but ignores structure: it can straddle a cut mid-interval, waste tiles on a static shot, or land mid-pan on a motion-blurred frame. Two refinements, both feeding --at. Choose by footage type: edited content (films, trailers, TV) → shot boundaries; continuously shot footage (handheld, drone, screen recordings, dashcam) → dwell points; unknown → uniform first, refine after.

Shot boundaries (cuts)

Detect cuts and align tiles to them:

# ffmpeg scene score: frames whose difference from the previous frame exceeds 0.3
TS=$(ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)',metadata=print:file=-" -f null - 2>/dev/null \
     | grep -oP 'pts_time:\K[0-9.]+' | paste -sd,)
python3 scripts/contact_sheet.py input.mp4 --at "$TS"

Each selected frame is the first frame of the new shot (the score compares against the previous frame). Threshold 0.3–0.4 suits most content; lower it for subtle cuts, raise it for noisy footage.

Know what scene detection misses. The scene score is a frame-pair difference metric:

• Gradual transitions (dissolves, fades, wipes) spread the change across many frames, each below threshold — they often go undetected.
• Within-shot content changes: between two cuts, objects can enter, leave, or change substantially with no boundary ever firing. A long static-camera shot with lots of action yields one tile under pure scene-aligned sampling.
• Camera motion (pans, handheld shake) can fire false positives.

So treat scene-aligned sampling as a refinement, not a replacement: run uniform sheets first for guaranteed temporal coverage, then a scene-aligned sheet (or a union of both timestamp sets via --at) when shot structure matters. If cut detection quality itself matters, the purpose-built tool is PySceneDetect (uv pip install --system scenedetect[opencv-headless], then scenedetect -i input.mp4 list-scenes) — its content/adaptive detectors are more robust to motion and noise than the raw ffmpeg score, but they are still cut-oriented and share the within-shot blindness above.

Dwell points (where the camera settles)

In continuously shot footage cuts are rare or absent — the structure lives in camera moves. In the frame-difference signal, held compositions are the valleys and pans/zooms are the peaks between them. The valley midpoints are the frames worth sampling: sharp, deliberately framed, one per composition — where uniform sampling would land mid-pan on smeared pixels.

python3 scripts/contact_sheet.py input.mp4 --at "$(python3 scripts/dwell_points.py input.mp4 --max 16)"

dwell_points.py computes the motion signal cheaply (4 fps at 160 px via ffmpeg's scene metric), smooths it over ~1 s, and takes spans below a relative threshold (default p40 of the signal) lasting at least --min-dwell (1 s). It keeps the --max longest holds and prints their midpoints; stderr lists each hold span with its duration so you can see the video's rhythm before reading a single frame.

Caveats:

• The threshold is relative, so on constant-motion footage (one unbroken pan) it degrades to picking the least-motion moments — harmless but arbitrary; prefer uniform sampling there. It exits non-zero when no hold lasts --min-dwell.
• Subject motion during a held shot (a person gesturing in a static frame) raises the valley floor but rarely fills it; if a busy-but-held shot is missed, lower --percentile.
• Dwells say where the camera settled, not what changed — combine with uniform coverage for anything time-critical.

Overseeing a multi-clip edit

When acting as the editing agent over a generated or assembled cut (see the creating-video skill), review at two levels:

1. The whole assembly. Contact-sheet the final stitched cut, not just the individual clips. Per-clip sheets each look fine in isolation; character drift, prop jumps, and logic breaks only appear when the shots sit in sequence. One 4×4 sheet over a 30 s cut gives ~2–3 tiles per scene — enough to catch them.

2. The seams. A cut hides continuity errors at the boundary — the outgoing clip's last frame vs the incoming clip's first frame. scripts/seams.py pairs them, one row per cut, for a one-look check:

   python3 scripts/seams.py clip1.mp4 clip2.mp4 ... clipN.mp4 --out seams.png
   

Read every sheet against the continuity checklist for generated video:

• Character — same face/hair/wardrobe across every shot they appear in.
• Prop — same identity, size, color, and attachment point shot to shot (the classic failure: a small object that changes size or moves between shots).
• Physical logic — is the world coherent across the cut (a window open before something enters through it; an action's result matching its setup)?
• Action completeness — is the key beat actually shown, not cut around?

Report which scene or seam fails and what the fix is (tighten the prompt, or regenerate just that shot) — that verdict is the deliverable the editing agent acts on.

Interpreting honestly

• Report what is visible in the sampled frames; events between samples are invisible. Say "between 1:30 and 1:40 the scene changes from X to Y", not fabricated specifics about the transition.
• Fast action (a ball in flight, a single gesture) can fall entirely between samples — tighten the window and re-sheet before concluding something didn't happen.
• Timestamps are accurate to ~1 s (seek + rounding), fine for general video.

Limits — when NOT to use contact sheets

• Fine print, dense text, small UI details: tiles are ~380 px wide; text becomes unreadable. Extract full-resolution frames at the relevant timestamps instead.
• Audio content: sheets are silent. If speech matters, extract the audio (ffmpeg -i in.mp4 -vn audio.mp3) and transcribe it separately; note to the user if no transcription path is available.
• Frame-exact analysis (sports officiating, VFX QC): sampling misses frames by design; extract every frame in a narrow window (ffmpeg -ss 84 -to 86 -i in.mp4 frames_%03d.png).

For transformation tasks — convert, trim, merge, compress, GIF, subtitles — use the processing-video skill.