/setup-video-vision

Setup Video Vision

Guide the user through configuring claude-video-vision step by step. Ask one question at a time using multiple choice. After each answer, proceed to the next step.

Step 1: Backend Selection

Ask the user:

Which backend do you want to use for audio analysis?

a) Gemini API (recommended) — Best quality. Analyzes audio natively with Gemini Flash. Free tier: 1500 requests/day. Requires a GEMINI_API_KEY (free at ai.google.dev).

b) Local (Whisper) — Free, fully offline. Uses whisper.cpp or openai-whisper for audio transcription. No cloud dependency.

c) OpenAI Whisper API — Good quality. Requires OPENAI_API_KEY. Paid per usage.

All backends use ffmpeg to extract video frames — Claude sees the frames directly.

After the user answers, call video_configure with the chosen backend.

Step 2: Whisper Configuration (only if backend is "local")

If the user chose Local, ask these questions one at a time:

Engine

Which whisper engine?

a) whisper.cpp (recommended) — Faster, less RAM, optimized for Mac/Linux

b) Python (openai-whisper) — More flexible, easier to extend

Call video_configure with whisper_engine.

Model

Which whisper model? Your system has [detect RAM with video_setup] of RAM.

a) tiny (75MB) — Very fast, basic quality

b) small (500MB) — Good balance of speed and quality

c) large-v3-turbo (1.5GB) — Best cost-benefit, recommended for 8GB+ RAM

d) large-v3 (2.9GB) — Maximum quality, recommended for 16GB+ RAM

e) auto — Let the plugin choose based on your hardware

Call video_configure with whisper_model.

Audio Tags

Enable Whisper-AT for non-speech audio detection? (coughs, music, animal sounds, etc.)

a) Yes — Detects non-speech events (requires Whisper-AT installed)

b) No — Speech transcription only

Call video_configure with whisper_at.

Step 3: Frame Configuration

Ask these one at a time:

Resolution

Frame extraction resolution (width in pixels, height auto-scales)?

a) 256px — Low res, fast, fewer tokens

b) 512px (default) — Good balance

c) 768px — Higher detail

d) 1024px — Maximum detail, more tokens

Call video_configure with frame_resolution.

FPS

Default frames per second extraction rate?

a) auto (recommended) — Adapts based on video duration (shorter = more frames, longer = fewer)

b) Custom value — Ask user for a number

Call video_configure with default_fps.

Frame Mode

How should Claude receive the frames?

a) Images (default) — Claude sees the actual frames (better perception, more tokens)

b) Descriptions — A sub-agent describes each frame as text (fewer tokens, loses visual nuance)

Call video_configure with frame_mode.

If descriptions mode:

Which model for the frame describer agent?

a) Sonnet (default) — Good balance

b) Opus — Most detailed descriptions

c) Haiku — Fastest, most concise

Call video_configure with frame_describer_model.

Step 4: Dependency Check

Tell the user:

Let me verify your setup...

Call video_setup with the configured backend and options. Show the results. Mention that yt-dlp is optional but required for YouTube URL support.

If dependencies are missing, show the installation commands and ask:

Want me to install these for you?

Step 5: Test (Optional)

After setup is complete, ask:

Setup complete! Want to test with a quick video? If so, provide a path to any video file.

If the user provides a video, call video_watch on it and show a brief summary of the results.

Important

• Ask ONE question at a time — never combine multiple questions
• Use the multiple choice format consistently
• Call video_configure after EACH answer to save incrementally
• If the user says "default" or "just use defaults", set all defaults and skip to Step 4
• If the user seems experienced, keep it concise — don't over-explain