Turning an Audiobook Into YouTube Shorts: The Full Pipeline

I had a finished audiobook. 47 chapters of generated TTS audio, fully verified, zero word loss. The book was live on KDP. The question was: how do I get pieces of it in front of people on YouTube Shorts?

The answer turned into a pipeline that takes an audiobook and a cover image and produces numbered vertical videos with burned-in captions — fully automated, no video editing software involved. Here is how it works.

The Input

Two things go in:

1. The audiobook audio file — one WAV per chapter, already generated and verified by the TTS pipeline.
2. The cover art — a high-resolution PNG, the same file used for the KDP paperback cover.

That is it. No video source material. No stock footage. No screen recordings. The video is built from static cover art and raw audio. This decision was deliberate: composited video is expensive to produce and irrelevant when the content is spoken word. Podcast clips on YouTube routinely use static backgrounds. The audio and the captions carry the content.

Slicing the Audio

YouTube Shorts have a 60-second limit. My chapters are 20-40 minutes long. I needed to slice each chapter into Short-sized segments.

The slicing logic uses silence detection. I run ffmpeg's silencedetect filter across the chapter audio to find natural pauses — breath points, scene breaks, paragraph endings. Then I greedily accumulate segments from those pause points until the next pause would push the segment past 55 seconds. The 5-second margin gives room for caption fade-in and fade-out without the video feeling cut off.

silence_detect(chapter_wav) -> pause_timestamps

for each pause_timestamp:
    if accumulated_duration + next_pause > 55s:
        emit segment
        start new segment

Not every segment comes out clean. Some chapters have long unbroken paragraphs with no pauses longer than 200ms, and the slicer produces a 50-second segment that ends mid-sentence. I flag those for boundary adjustment — nudging the cut point to the nearest sentence end. The rule is: cut at sentence boundaries, never mid-sentence. A Short that ends on a complete thought gets rewatched. One that cuts off gets swiped away.

WhisperX Word Timestamps

Each audio segment then goes through WhisperX forced alignment. I feed it the segment audio and the exact manuscript text for that segment. WhisperX returns word-level timestamps — the start and end time of every individual word in the audio.

These timestamps are the backbone of the caption system. Without them, I would be guessing at timing. With them, I know exactly when each word is spoken and can display captions that are perfectly synced.

I have written separately about WhisperX's deterministic blind spots and the midpoint-split repair that closes 85% of alignment gaps. That repair runs as part of this pipeline — if WhisperX returns null timestamps for any words, the gap repair script kicks in before caption generation.

Three-Zone Caption Layout

Vertical video (1080×1920) has a specific reading challenge. The viewer is watching on a phone, probably in a distracted context, and the text needs to be immediately readable. I use a three-zone layout:

• Top zone (0-15%): Title and byline. The book title and author name, always visible as context for what the viewer is watching.
• Middle zone (30-60%): A catchy hook pulled from the caption text. Two lines maximum, large font, high contrast outline. This is where the viewer's eye naturally sits on a phone screen.
• Lower zone (75-95%): The full caption text. Readable but secondary — supporting detail for anyone who wants to follow along word by word.

The lower zone carries the detail captions, but the middle zone is the hook — the short phrase that stops someone from scrolling past. The top zone provides permanent context: what is this and who made it. Three zones, three jobs.

The caption text itself uses word-grouping logic. Rather than showing one word at a time (karaoke style) or the full sentence (subtitle style), I show 2-4 word phrases grouped by natural speech patterns. WhisperX's timestamps make this precise — I group words based on their timing gaps, where a gap of 200ms or more indicates a natural phrase boundary.

Portrait ASS Generation

The caption data — word groups with timestamps — gets compiled into an ASS (Advanced SubStation Alpha) subtitle file. ASS is the same format used for anime subtitles, and it supports precise positioning, styling, and timing control.

The generator writes ASS events with:

• Screen position locked to the middle zone
• Font size scaled for 1080px width
• Bold white text with a black outline for contrast against any cover art background
• Fade-in and fade-out transitions (150ms each)
• Sequential display so each word group appears, holds, and disappears before the next

This ASS file is then burned into the video using ffmpeg's subtitles filter — the same approach I describe in the 60x speedup post. One ffmpeg pass, no intermediate frame rendering.

Assembling the Video

The final video for each Short is:

1. Cover art scaled to 1080×1920 (cropped, not stretched, to preserve aspect ratio).
2. Raw audio segment as the audio track.
3. ASS captions burned in via the ffmpeg subtitles filter.

ffmpeg -loop 1 -i cover.png -i segment.wav \
  -vf "subtitles=captions.ass,crop=1080:1920" \
  -c:v libx264 -tune stillimage -c:a aac -b:a 192k \
  -shortest output.mp4

The -tune stillimage flag tells x264 to optimize for a video that is essentially a single frame. This dramatically reduces file size without quality loss, since there is almost no inter-frame motion to encode.

Each Short takes about 0.8 seconds to render. A full 47-chapter book with roughly 300 segments processes in under 5 minutes.

Sequential Numbering

Every Short gets a sequential number in its filename and metadata: short_001.mp4, short_002.mp4, and so on. This is not just for organization — it is the upload order. YouTube's algorithm treats the first few videos in a new channel's batch upload as context for what comes next. Sequential numbering ensures that a viewer who finds Short #047 can trace back to Short #001 and follow the narrative.

The number is also baked into the title: "The Incubator — Ch.3 Pt.2 (47/300)". This tells the viewer three things: which book, which chapter, and where they are in the sequence. Metadata in the title is free real estate. Use it.

What I Left Out

Things I considered and rejected:

• Background music. The audiobook audio is clean TTS. Adding music would compete with the voice and complicate the mixing. The silence between words is part of the pacing.
• Animated cover art. Ken Burns zoom effects on the cover image. Looks nice on a demo, adds zero value for a viewer who is there for the story. And it increases render time 10x.
• AI-generated visuals. Scene-by-scene image generation to accompany the narration. This is the obvious "wouldn't it be cool" feature, and it is a time sink that does not improve the core product: the writing, delivered as speech, with readable captions.

The pipeline works because it does exactly what it needs to and nothing more. Cover art, audio, captions. Every additional feature is a maintenance burden and a render-time cost that has to justify itself against the one metric that matters: does the viewer finish the Short?

The Output

For a 47-chapter book, the pipeline produces around 300 vertical videos. Each one is a 40-55 second clip from the audiobook with synchronized captions over the book's cover art. They are numbered, titled, and ready for batch upload to YouTube.

The whole thing runs from a single command after the audiobook is finalized. No manual video editing. No design work. No ongoing maintenance between books — the same pipeline works for any audiobook with a cover image.

That is the pipeline. Static image, raw audio, word-level captions, vertical format. It is not fancy. It produces exactly what it needs to produce, and it does it fast enough that the bottleneck is uploading 300 videos to YouTube, not generating them.