WhisperX Has Deterministic Blind Spots (And How to Fix Them)

When you run the same audio through WhisperX twice, you get the same gaps. Not approximate gaps — the exact same words without timestamps at the exact same positions. This is not random failure. It is deterministic failure, which means you can detect it, isolate it, and fix it.

I discovered this building the caption pipeline for my audiobook project. The pipeline generates audiobook audio from a manuscript using TTS, then runs WhisperX forced alignment to pin every word in the manuscript to its exact timestamp in the generated audio. Those word-level timestamps drive automated subtitles — ASS files burned into video with ffmpeg. When words show up as unaligned (null timestamps), they vanish from the captions. The viewer sees a sentence with holes in it.

I assumed WhisperX was occasionally dropping things — stochastic noise in a model that is mostly accurate. I re-ran the failed segments. Same gaps. Same words. Same timestamps. Every time.

That changed the problem entirely. Random errors need retries. Deterministic errors need a different strategy.

Building planning-first AI systems, one pipeline at a time. Get weekly deep dives into local AI, Rust, and agent orchestration — no fluff, no hype.

The Pattern

WhisperX forced alignment works by taking a transcript and an audio file and using cross-attention to pin each word to a time region. It is not transcribing freely — it is aligning known text to audio. Most of the time this produces precise word boundaries. But certain audio regions consistently defeat the aligner — usually ones with unusual prosody, breath pauses, or phonetic ambiguity in the generated speech.

The blind spots follow a pattern:

  • They are reproducible. Run the same file 10 times, get the same 10 gap sets. No variance. Different seed, different batch size, same result.
  • They cluster at segment boundaries. Most gaps appear near where Whisper's internal segmentation split the audio. The aligner loses the boundary word.
  • They are word-level, not segment-level. WhisperX accepts the full transcript but returns null timestamps for individual words. The word is in the text; the timing is not.

If your caption pipeline is missing 2% of words and that 2% is identical on every run, retrying is useless. You will burn compute re-running the exact same failure. The fix is not more attempts — it is a different approach to the same audio.

Why Retries Do Not Work

The instinct when something fails is to try again. Maybe the model was tired. Maybe the batch hit a bad state. This works for stochastic systems. WhisperX's alignment is not stochastic in any meaningful sense for these blind spots.

I tested this. I took a chapter with 14 known gaps and ran it through WhisperX five times with different batch_size values (4, 8, 16, 24, 32). The gaps were identical across all five runs. Same word indices, same null timestamps, same everything. The batch size changed processing time but did not change the output.

I also tried varying the compute type (float16 vs int8). The gaps moved by 1-2 words in some cases, but 11 of the 14 gaps were present across all configurations. The alignment model has specific audio features it cannot resolve, and no amount of parameter tuning changes that.

Retry logic is the wrong tool for deterministic failure. It burns time and compute without changing the result. What you need is a repair strategy.

The Midpoint Split Repair

The fix is counterintuitive: cut the audio in half at the gap and align each half separately. WhisperX's blind spots are tied to the full segment's alignment context. When you split the audio, the segment boundaries change, and the aligner processes the problematic region with different context — context that happens to work.

Here is the repair script in pseudocode:

for each gap in whisperx_output:
    gap_start = gap.segment_start
    gap_end   = gap.segment_end
    midpoint  = (gap_start + gap_end) / 2

    # Split the WAV at the midpoint of the gapped segment
    part_a = wav[segment_start : midpoint]
    part_b = wav[midpoint : segment_end]

    # Re-run WhisperX on each half independently
    result_a = whisperx.align(part_a)
    result_b = whisperx.align(part_b)

    # Merge the results back into the full timeline
    merge(result_a, result_b, offset=segment_start)

The actual implementation is a Python script that reads the WhisperX JSON output, extracts the segment audio via ffmpeg, runs two subprocess calls to whisperx_align.py, and stitches the word timestamps back into the master alignment file. Each gap repair takes about 4 seconds on my machine. A chapter with 14 gaps takes under a minute.

The key insight is that the midpoint split changes the alignment context without changing the audio content. The same words, the same audio waveform, the same model — but different segment boundaries mean the cross-attention heads resolve differently. The blind spot was an artifact of how the audio was chunked, not what was in it.

Results

I ran this repair across the full audiobook — 47 chapters, 312 total gaps. After one pass of midpoint splitting:

  • 265 gaps resolved (85%). Words that were consistently missing across every previous run now had clean timestamps.
  • 34 gaps partially resolved. Some words appeared but with low-confidence timing. Usable, but flagged for manual review.
  • 13 gaps remained. These were in audio with heavy background music or unusual acoustic features. The midpoint split did not help because the underlying audio genuinely confuses the aligner.

85% gap closure from a single repair pass is significant. It took the caption output from sentences with visible holes to fully readable subtitles — and the remaining gaps are consistently the same ones, so they can be manually verified once and cached.

The repair script is idempotent. Running it twice on the same gap set produces the same result, because the midpoint split is deterministic. This matters for pipeline recovery — if the repair step crashes halfway through, re-running it does not create conflicting alignments.

When to Stop Repairing

The remaining 13 gaps — the ones that survived midpoint splitting — could theoretically be fixed with finer-grained splits. Quarter the audio instead of halving it. Try different split points. But there are diminishing returns.

At 85% closure, the caption pipeline is producing clean subtitles. The 15% that remain are genuinely hard audio regions where the alignment model fails for acoustic reasons, not segmentation reasons. Splitting those further might recover a few more, but the cost — more subprocess calls, more merge logic, more edge cases in the timestamp stitching — is not worth the 2-3 additional words per chapter.

The repair should stop when the remaining gaps are caused by the audio, not the chunking. Midpoint splitting fixes chunking artifacts. It cannot fix a model that genuinely cannot align a specific acoustic pattern. Knowing the difference is what keeps the repair script simple.

The Principle

Deterministic failure is easier to fix than stochastic failure, but only if you recognize it. The trap is treating deterministic failure as if it were random — retrying, varying parameters, hoping for a different result. The WhisperX blind spots looked like flaky model behavior until I re-ran the same file and got byte-identical output. That is when the problem became tractable.

The general pattern: when a failure is reproducible, the fix is not retry — it is recontextualizing the input. The same operation on a different view of the same data often succeeds. Midpoint splitting is one instance of this. You will find the same pattern anywhere a chunk-based system has boundary artifacts.

If your pipeline has a step that consistently fails on the same inputs, stop retrying. Check whether the failures are deterministic. If they are, change how the input is presented to the system. The model is not broken. It just needs a different angle on the same data.