Turn a movie into one dense, agent-friendly markdown brief.
mdai takes a movie (URL or local video file), pulls the transcript and timestamped frames, packages them into timeline chunks, feeds them through a vision-capable AI agent, and emits a single comprehensive markdown document — structured for consumption by another AI agent rather than a human reader.
gem install mdai| Dependency | Required for | Install |
|---|---|---|
ffmpeg / ffprobe |
frame extraction, subtitle probing (always) | brew install ffmpeg |
yt-dlp |
URL sources and their captions | brew install yt-dlp |
claude CLI |
the default claude-cli agent |
https://claude.com/claude-code |
CLAUDE_CODE_OAUTH_TOKEN |
headless claude -p auth |
claude setup-token |
ANTHROPIC_API_KEY |
the anthropic agent |
https://console.anthropic.com |
| transcriptor | whisper transcription fallback (optional) | path/git dependency |
Run the doctor to see what's available:
mdai deps# Local file, default agent (claude CLI), output ./<slug>.md
mdai movie.mp4
# URL source with the Anthropic API agent
mdai https://example.com/watch?v=xyz --agent anthropic --model claude-sonnet-5 -o brief.md
# Frames-only (skip transcript entirely)
mdai movie.mp4 --frames-only
# Tighter budget: fewer frames
mdai movie.mp4 --max-frames 60source ──> acquire ──> transcript ──> frames ──> chunk ──> describe ──> synthesize ──> render
(probe) (tiered) (ffmpeg) (pure) (map: agent (reduce: one (markdown)
per chunk) final pass)
Every stage is idempotent against a cached workspace (~/.cache/mdai/<id>/ by default) —
re-running skips completed stages, --force redoes them. Agent calls are the expensive
part; the cache means you never pay for the same chunk twice.
- embedded — subtitle track inside the file (ffprobe/ffmpeg)
- ytdlp — subtitles or auto-captions for URL sources (yt-dlp)
- whisper — local transcription via the optional transcriptor gem
- none — degrades to a frames-only document with an explicit warning section
--transcript auto (default) tries them in order; an explicit value forces one source.
Adapters are thin: complete(prompt:, images:) -> String. Prompts live outside the
adapters, so adding OpenAI/ollama later is a one-file change.
claude-cli(default) — shells out to theclaudeCLI in headless mode; vision via image file paths. Zero API-key setup; respects your CLI's configured model.anthropic— Anthropic Messages API overnet/http(no SDK dependency); base64 image blocks; retries on 429/5xx. Default modelclaude-sonnet-5(1M context — ample for this workload); pass--model claude-opus-5for the harder reasoning tier.
The two agents bill differently, which is easy to miss. claude-cli runs
against your Claude subscription (via claude setup-token) — no per-token
charge. anthropic uses your ANTHROPIC_API_KEY and is metered API
pricing. A ~5-minute video at --max-frames 30 --chunk-frames 8 runs about
20K input / 5K output tokens — roughly $0.20-0.25 on Opus, less on Sonnet.
| Flag | Default | Purpose |
|---|---|---|
-o, --output PATH |
./<slug>.md |
final markdown path |
--agent NAME |
claude-cli |
claude-cli or anthropic |
--model NAME |
adapter default | passed through to the agent |
--transcript MODE |
auto |
auto, embedded, ytdlp, whisper |
--frames-only |
off | skip transcript entirely |
--frame-strategy MODE |
auto |
auto/fusion, transcript, or scene |
--concurrency N |
1 |
describe N chunks at a time (drops carry-forward) |
--keep-duplicates |
off | skip perceptual dedup of near-identical frames |
--scene-threshold T |
0.4 |
ffmpeg scene-change sensitivity (scene strategy) |
--interval N |
off | force one frame every N seconds instead of scene-detect |
--max-frames N |
sensed from duration | override the frame budget |
--single-pass / --no-single-pass |
auto | force or forbid one-call mode |
--single-pass-max-frames N |
40 |
fit threshold for one-call mode |
--chunk-frames N |
20 |
max frames per agent call when chunking |
--chunk-window N |
600 |
max seconds of timeline per chunk |
--prepare-only |
off | stop before agent calls; write prompts + chunk plan for review |
--workdir PATH |
~/.cache/mdai/<id> |
workspace/cache override |
--agent-timeout N |
900 |
seconds before a hung agent call is killed |
--whisper-model NAME |
small |
whisper model for the transcriptor backend |
--force |
off | redo all cached stages |
--verbose / --quiet |
— | more / less progress output |
The output file is written atomically and overwrites an existing file at the
same path (default ./<slug>.md). Changing content-affecting flags
(--max-frames, --model, --transcript, ...) automatically invalidates and
re-runs just the affected pipeline stages — --force is only needed to redo
everything from scratch.
---
title: ...
source: ...
duration: HH:MM:SS
generated: <ISO8601>
agent: claude-cli
model: ...
frames: 142
transcript_source: embedded
---
# Title
## Overview
## Characters & Speakers
## Timeline
### [00:00:00–00:09:58] Scene label
**Visual:** ...
**Dialogue:** ...
## Full Transcript (collapsible appendix)Every intermediate artifact is kept in the workspace (default
~/.cache/mdai/<id>_<slug>/):
meta.json duration/title/video path
video.* the downloaded video (URL sources)
transcript.json normalized transcript + which source produced it
frames/ every extracted jpeg + index.json (timestamps)
frames/delta_curve.json the whole-video visual change curve
frame_plan.json which moments were chosen, and why
chunks.json the chunk plan (windows, frame/segment counts)
prompts/ the EXACT prompt sent to the agent, per call
summaries/ the agent's per-chunk descriptions
synthesis.md the reduce-step output
config/ per-stage configs (drive cache invalidation)
Two tools for working with them:
# Stage status table + chunk plan for a source's workspace
mdai inspect movie.mp4
# Run everything UP TO the agent calls, dump all prompts, and stop -
# review prompts/ and frames/, tune flags, then run for real (the
# prepared stages are all cached)
mdai movie.mp4 --prepare-onlyTwo signals answer different questions, and mdai fuses them by default
(--frame-strategy auto):
- Visual delta — "the picture changed." Catches events with no verbal cue: a diagram appears, a UI transition, a cut to new footage. Cheap, no LLM, covers the whole timeline.
- Transcript salience — "this moment matters." Catches importance with no visual cue: "as you can see here", topic transitions, demos. One text-only agent call.
Neither alone is enough. A talking head has near-zero visual delta but may carry the key claim; a silent screencast has no transcript signal but constant visual change. Fusion does four things with them:
- Snap to stable — a cue at 4:57 lands on a settled frame, not a motion-blurred transition.
- Merge and rank — moments both signals nominate rank first when the budget is tight.
- Budget by density —
--max-framesis distributed toward the information-dense stretches. - Guarantee coverage — no stretch of the video goes entirely unseen.
Near-identical frames are then dropped perceptually, so a 40-minute talking head can't burn the whole budget on one face.
Every choice is recorded in frame_plan.json with its reasoning, and each
frame's provenance travels into the prompts — so the reading agent knows
why it's looking at a given frame.
Other strategies: --frame-strategy transcript (cues only, no delta scan)
and scene (the legacy scene-detect/interval path).
Because absolute thresholds are the wrong instrument for unedited content. Measured on a real 5-minute screencast:
| result | |
|---|---|
scene detect at every threshold 0.4 → 0.05 |
0 frames → blind 3-second sampling, 101 frames |
| fused delta curve, budget 12 | 12 frames, including the title cut and the key mid-video state change |
The signal was always there — a 100× dynamic range with clear local peaks — it just never approached cut-shaped magnitudes. Scanning at 2fps also makes gradual transitions visible: a fade is invisible frame-to-frame but obvious across half-second samples.
You shouldn't have to tune anything for a normal run. mdai <url> reads the
video's duration and sizes the work to it — the frame budget scales
sub-linearly, so coverage grows while density falls, the way you'd actually
skim longer material:
| Duration | Frames |
|---|---|
| 1 min | 13 |
| 5 min | 30 |
| 15 min | 52 |
| 30 min | 73 |
| 1 hour | 104 |
| 2 hours | 147 |
A two-hour film gets 5× the frames of a five-minute clip, not 24×. Every flag below remains available to override the sensed value when you need to.
Short movies are described in a single call — every frame and the full transcript go up together, and the model writes the document directly. Nothing is squeezed through a per-chunk summary first, so observations that span the whole film survive.
Longer movies fall back to map/reduce: describe each timeline window, then
synthesize. That exists to bound per-image attention and to keep a long run
resumable when one call fails — real benefits at 100+ frames, pure overhead at
30. The switch is the frame count (--single-pass-max-frames, default 40);
override it either way with --single-pass / --no-single-pass.
The map phase is embarrassingly parallel. --concurrency N describes N chunks
at a time, trading the carry-forward chain (each chunk seeing a summary of
what came before) for wall-clock. The synthesis pass still sees every chunk,
so the loss is bounded to cross-chunk callbacks.
The dominant cost is the agent, not the pipeline. The anthropic agent
sends a whole chunk's frames in one request. The claude-cli agent reads
images one per turn, so a 20-frame chunk is a multi-minute call — attacking
that with concurrency treats the symptom, and running several CLIs at once
has proven unreliable in practice. For frame-heavy runs prefer
--agent anthropic, and lower --chunk-frames.
Tuning rules of thumb:
- Visually repetitive content (simulations, screencasts, lectures): let
fusion do its job and cut the spend —
--max-frames 30 --chunk-frames 8. - Fast cuts / action: raise
--max-frames; the delta curve will find the cuts on its own. - The claude-cli agent reads images one per turn, so a 20-frame chunk is a
multi-minute call (heartbeat lines show progress). The
anthropicagent sends all of a chunk's frames in a single request — much faster for frame-heavy runs. - Fast cuts / action: raise
--max-frames, lower--scene-threshold(e.g. 0.3) so more cuts survive.
bundle install
bundle exec rake # specs
bundle exec rubocop # lint
MDAI_INTEGRATION=1 bundle exec rspec --tag integration # opt-in ffmpeg integration specsMIT. See LICENSE.txt.