60x Speedup: Why ffmpeg's subtitles Filter Beat My Custom Caption Burner
The YouTube Shorts pipeline needed burned-in captions. Words on screen, synced to audio, styled for vertical video. I had ASS subtitle files with word-level timestamps from WhisperX. I had cover art frames. I had ffmpeg. The question was how to combine them.
I chose to build a custom PNG overlay pipeline in Rust. Three weeks later, I threw it away and used a single ffmpeg filter.
The Custom Pipeline
The design seemed reasonable at the time. I was already generating cover art frames as PNG files. The caption text came from ASS files with precise word-level timing. The plan:
- Decode each video frame to a raw image in memory.
- Render the caption text onto a transparent PNG at the right timestamp.
- Composite the caption PNG over the video frame.
- Re-encode the composited frame back into the video.
This ran in Rust, using a worker pool to parallelize frame processing. Each worker grabbed a frame, rendered its caption overlay, and wrote the result. It felt like the right approach — I had control over every pixel, I could customize the text layout, and I was using the language I was already working in for the rest of the pipeline.
The problem was speed. Or rather, the complete absence of it.
At 1.8 frames per second, a 60-second Short took 33 seconds to process. That sounds almost acceptable until you need to render 400 of them. That is over three hours of processing for captions alone. And the pipeline was not just slow — it was fragile. Frame decoding deadlocked under memory pressure. The worker pool would stall when one worker hit a large frame. PNG compression was eating CPU on every single overlay, even when the caption text had not changed from the previous frame.
I spent a week optimizing. I tried pre-rendering caption strips and blitting them onto frames. I tried reducing the overlay resolution. I tried fewer workers, more workers, different PNG encoders. The best I achieved was 3.2 fps. Still 18 seconds per Short. Still unacceptable for batch processing.
The Discovery
While debugging a separate ffmpeg issue — an audio sync problem in the concatenation step — I stumbled onto the subtitles filter. It is part of libass, the same library that handles ASS parsing in every major video player. ffmpeg wraps it natively.
The filter takes an ASS file and burns it directly into the video stream during encoding. No intermediate PNGs. No frame-by-frame compositing. No worker pool. One command:
ffmpeg -i input.mp4 -vf "subtitles=captions.ass:force_style='FontName=Arial,FontSize=24'" -c:a copy output.mp4
I ran it on the same 60-second Short that my custom pipeline had choked on.
It finished in 0.56 seconds. 107 fps.
Sometimes the tool you need is already installed. You just have not read far enough into the documentation.
Why It Was Faster
The performance difference was not magic. It was architecture.
My pipeline decoded every frame to raw pixels, composited in a separate process, then re-encoded. That means three passes over every frame: decode, composite, encode. Each pass crosses a process boundary or a memory allocation boundary. The PNG overlay approach added a fourth pass: PNG encode and decode for the caption layer.
ffmpeg's subtitles filter runs inside the encoder pipeline. It operates on frames as they pass through the filter graph — decode, render text on top, encode, all in a single pass. libass handles the text rendering with heavily optimized sub-pixel rasterization. ffmpeg handles the frame I/O without ever materializing a separate overlay image.
The 60x speedup was not from a clever algorithm or a better implementation of the same approach. It was from eliminating the approach entirely. The fastest code is the code that does not run.
When Custom Makes Sense
The irony is that the custom pipeline was not wasted effort. It taught me exactly what I needed: the three-zone text layout for vertical video, the word-grouping logic for readable captions, the timing adjustments for when captions appear and disappear. All of that knowledge went into the ASS files that ffmpeg now renders.
The difference is that the ASS file describes what to render. ffmpeg handles how to render it. My custom pipeline tried to do both, and the "how" was where all the time went.
Define your intent in a declarative format. Let a dedicated tool execute it. This is the same principle as writing a config file instead of a script — except here the config file is ASS and the tool is libass.
There are cases where a custom renderer is the right call. If you need captions that respond to audio amplitude in real time, or captions that interact with on-screen objects, or captions rendered in a style that libass does not support — then you need custom code. But for standard styled text overlays synced to timestamps, libass is faster, more correct, and more battle-tested than anything you will build in a week.
The General Principle
This is a specific instance of a pattern I keep running into: building infrastructure that already exists is the most expensive way to learn what it does.
I could have started with the ffmpeg subtitles filter, discovered its limitations, and only built custom code for the parts it could not handle. Instead, I started with custom code, discovered its limitations, and then found that ffmpeg handled everything anyway. The order matters. Start with the existing tool. Hit its actual limits, not its imagined ones. Only then build.
The temptation to build is strong, especially when you are already deep in a codebase. I had a Rust workspace, a worker pool, and a frame processing pipeline. Adding a caption renderer felt like a natural extension. It was not. It was a detour through a problem that someone else had already solved, better, inside a tool I was already using.
Three weeks of development. One ffmpeg flag to replace all of it. The 60x speedup was nice. The real lesson was knowing when to stop building.
