RELATED 01
claude-real-video
Turns a video link or local file into keyframes and a timestamped transcript an assistant can read
2.3kGitHub stars
3.4k/moPyPI downloads
64popularity score
claude-video-vision
Video perception for Claude Code: ffmpeg frames plus timestamped audio transcripts
By Jordan Vasconcelos (individual) TypeScript Local · stdio MediaAI and ML
63 popularity score ★ 1.5k ↓ 2.2k/wk
Connect your client
Local (npm)
Add in one click
Opens the app, which asks before it adds the server.
Or add it by command or config
Claude CodeTerminal
claude mcp add claude-video-vision -- npx -y claude-video-vision@latestClaude Desktopclaude_desktop_config.json
{
"mcpServers": {
"claude-video-vision": {
"command": "npx",
"args": [
"-y",
"claude-video-vision@latest"
]
}
}
}Cursor~/.cursor/mcp.json
{
"mcpServers": {
"claude-video-vision": {
"command": "npx",
"args": [
"-y",
"claude-video-vision@latest"
]
}
}
}VS Code.vscode/mcp.json
{
"servers": {
"claude-video-vision": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"claude-video-vision@latest"
]
}
}
}Codex~/.codex/config.toml
[mcp_servers.claude-video-vision]
command = "npx"
args = ["-y", "claude-video-vision@latest"]Gemini CLITerminal
gemini mcp add --scope user claude-video-vision npx -- -y claude-video-vision@latestSettings 2
GEMINI_API_KEY |
Optional: API key for the Gemini audio backend; needed only when backend is gemini-api. |
|---|---|
OPENAI_API_KEY |
Optional: API key for the OpenAI Whisper backend; needed only when backend is openai. |
What it does
claude-video-vision is a Node.js MCP server, shipped mainly as a Claude Code plugin, that turns a video into material Claude can look at. It pulls still frames out of a local file or a YouTube link with ffmpeg and returns them as base64 images (or as text descriptions), alongside a timestamped transcript of the audio. It does no interpreting of its own: the frames and transcript go to Claude, which answers the question.
Audio goes through one of three backends you pick: Google's Gemini API (which also tags non-speech sounds), Whisper running locally through whisper.cpp or Python openai-whisper, or OpenAI's Whisper API. For YouTube links it downloads the video with yt-dlp and prefers English manual subtitles, then automatic captions, and only falls back to the audio backend when captions are missing or too thin, labelling which source each transcript came from. Frame rate, time range, resolution and per-segment settings can be set on each call, so a question about one second can be answered at full frame rate and an hour-long lecture at a low one.
Settings live in ~/.claude-video-vision/config.json. In the v1.4.0 code the backend starts as "unconfigured" rather than the "local" shown in the README's example, and until a backend is chosen video_watch returns a setup message instead of results, unless audio is skipped or YouTube captions cover the video (mcp-server/src/config.ts, mcp-server/src/tools/video-watch.ts). The same file sets the Gemini audio model to gemini-3-flash-preview and splits audio longer than 1200 seconds into 600-second chunks by default.
What it needs
| Runtime | Node.js 24+ |
|---|
Node.js 24 or newer and ffmpeg on PATH. yt-dlp is needed only for YouTube links. For audio, one of: a Gemini API key (free tier available), a local whisper.cpp or openai-whisper install (models download on first use to ~/.claude-video-vision/models/), or a paid OpenAI API key. The plugin install route needs Claude Code; the npm command works in any MCP client.
Limits
Nothing comes back from video_watch until a backend is set with the setup wizard or video_configure, apart from frames-only calls and YouTube videos with enough captions. Frames are capped at 100 per call by default (max_frames) and frame width must be 128 to 2048 px. YouTube caption lookup prefers English tracks. The README notes that on Windows a broken or old npx can stop the plugin's server from starting, and gives a global npm install as the workaround; CI runs only on Ubuntu and macOS.
Who it suits
Suits Claude Code users who want Claude to inspect screen recordings, bug-report clips, tutorials or YouTube videos without uploading the whole file to a video service. The local Whisper backend keeps audio processing on your machine, at the cost of installing whisper and downloading models; the Gemini and OpenAI backends are less setup but send audio to those vendors. It is a community project, not from Anthropic, Google or OpenAI. Because it returns images as base64, long videos at high frame rates use a lot of context; pick low fps or view_sample for long material.
Tools 6
video_watch- Extracts frames from a file or YouTube URL and transcribes its audio, with fps, time range and segment options
video_analyze- Inspects a video's structure with ffmpeg filters before any frames are extracted
video_detail- Zooms into particular moments, from cache or by extracting them again
video_info- Returns a video's metadata without extracting anything
video_configure- Changes the saved settings such as backend, frame format and limits
video_setup- Checks for ffmpeg, whisper and other dependencies and explains how to install missing ones