Watch Skill

What it does

Watch Skill is a local Python server that gives an assistant a way to take in video. It downloads a source (any site yt-dlp handles, a direct media or HLS/DASH link, or a local file), keeps scene-aware frames, reads on-screen text, transcribes speech from captions or local Whisper, and stores it all in an index on your machine. Follow-up questions, moment lookups and searches across every video watched so far are then answered from that index without processing the video again.

It goes further than playback analysis. It can record new footage of a web page, the desktop or a named window; watch a live page or stream and answer questions about what just happened; and run a capture, critique, fix, re-capture cycle on a UI, a game or a generated video. A separate verify_contract tool decides whether an agent run worked using fixed checks (a file exists, a JSON value, an HTTP response, a command's exit code) rather than a model's opinion, and keeps a hash-bound evidence record.

From config.py at v1.4.3: everything lives under ~/.watch-skill unless WATCHSKILL_DATA_DIR says otherwise, the download cache is capped at 20 GB by default with least-recently-used eviction, a tool response carries at most 12 images by default (WATCHSKILL_RESPONSE_FRAME_CAP), and cloud vision spending is capped by default at $1 per call and $5 per run. The default vision provider is Anthropic. The same package also offers a CLI and a REST API, and a separate DeepWatch workspace is built on top of it.

What it needs

Python 3.11, 3.12 or 3.13 and uv for the uvx command. ffmpeg and yt-dlp: `watch-skill doctor` (or the doctor tool) downloads them. On-screen text reading needs the [ocr] extra, local transcription the [whisper] extra, and browser capture and the loop tools the [loop] extra plus `playwright install chromium`. Answers and loop critiques that need a vision model need a provider API key (Anthropic by default) or a local Ollama. Windows, macOS and Linux.

Limits

The documented command installs only the [standard] extra: frames, retrieval and MCP. Without [ocr] it cannot read on-screen text, and a bare install without [standard] cannot extract a frame at all. Without a vision-capable model the loop's critique reports that it cannot judge; capture, frames, OCR and verification still work. Sending audio to a cloud speech service is off unless WATCHSKILL_CLOUD_STT_ENABLED is set, and cloud scene descriptions are not turned on just because a key is set. With WATCHSKILL_OFFLINE set it makes no network calls, so remote URLs fail unless already cached. Screen, window and camera capture depend on the machine; capture_capabilities reports what works.

Who it suits

Suits developers who want an assistant to work from videos, tutorials, meeting recordings or bug screen recordings, or to check its own UI changes against a recording. Video stays on your machine; frames go to a cloud model only when you set a provider key and the mode allows it. It is a heavy install with optional extras and outside binaries, and some tools run commands and record the screen, so it is a poor fit if you only need a transcript of a single video or want a server with no write or command access.