Auto B-Roll

An AI editor that watches a talking-head video, finds matching stock B-roll, and cuts in footage and captions automatically

The Problem

The first end-to-end version ran, but the output felt off — five compounding design flaws, not bugs. It searched on generic one-word keywords, always grabbed the first stock clip, and gave a random 50% of segments B-roll, so the same input produced a different edit every run. It was text-only and never 'saw' the footage it chose, and it made hard cuts with no captions. The pipeline worked; the editing decisions didn't.

The Solution

A re-architected pipeline with editorial logic baked in. Gemini 2.0 Flash analyses the actual video (not just the transcript) to target a broadcast-standard 65–75% B-roll coverage, leaving the speaker on screen for emphasis. Prompt engineering forces specific verb-and-noun search phrases ("hands adjusting an analog mixing console", not "music"), each with a confidence score. The top five stock candidates are ranked on resolution, duration and relevance instead of taking the first hit, and a multi-factor scoring model across temporal zones replaces random placement, so the same input always produces the same edit. Burned-in captions plus a portable .srt finish it for sound-off social viewing. Whisper handles transcription, GPT-4o is a text-only fallback, and MoviePy + FFmpeg assemble the cut.

How It Works

From a raw talking-head take to a broadcast-style cut — automatically.

Most creators shoot themselves talking to a camera, then lose hours hunting for B-roll and typing captions to make it watchable. This pipeline does that editing pass automatically: it transcribes the audio, uses a multimodal AI to decide which moments need visuals and what those visuals should be, sources the footage, and assembles a finished, captioned cut.

What started as a "weird and random" research notebook became a deterministic, broadcast-style editor — the real work was teaching it judgment, not writing more code.

Left: the raw talking-head input. Right: the AI's finished cut — B-roll and captions added automatically.