Designing Resume Logic: Queued vs Completed and Nothing Else

The narrator-tts pipeline processes a 120,000-word novel in roughly 800 segments. Each segment is a few sentences of text sent to a local TTS model, producing a WAV file. The full audiobook is the concatenation of all 800 segments.

A run takes about forty minutes. During that time, things go wrong. The GPU driver hiccups. A worker process crashes. The machine goes to sleep. Power drops.

When the pipeline restarts, it needs to pick up where it left off. That means answering one question for every segment: is this segment done?

The obvious answer is to track state. The wrong answer is to track too much state.

The State Machine I Almost Built

Building planning-first AI systems, one pipeline at a time. Get weekly deep dives into local AI, Rust, and agent orchestration — no fluff, no hype.

My first instinct was to give each segment a lifecycle:

pending → in_progress → completed
                 ↘ failed
                      ↘ retried → in_progress

Five states. Transitions between them. A manifest tracking which segment is in which state. Retry counters. Failure reasons. Timestamps for each transition.

This feels right because it mirrors how we think about work. You have not started yet. You are working on it. You finished. You tried and it broke. You are trying again.

Here is the problem: every state between "not done" and "done" is a liability.

The Problem With In-Progress

The in_progress state is the seductive one. It feels necessary. You want to know what the system is doing right now. You want to show a progress dashboard. You want to detect stuck workers.

But in_progress creates a race condition. A worker claims a segment, marks it in_progress, and starts generating audio. The worker crashes. The segment is now stuck in in_progress forever. No other worker will pick it up because it looks like someone is handling it.

You need a stale detection system. How long is too long for in_progress? Two minutes? Ten? The answer depends on your hardware, your model, your segment length. Get it wrong and you either retry segments that are still processing (producing duplicate work) or you wait too long to retry stuck segments (producing delays).

Every intermediate state is a failure recovery problem disguised as a feature.

The failed state is worse. Now you need retry logic. How many retries? With what backoff? Do you escalate to a different worker? Do you flag it for manual review? Each answer adds complexity.

Two States. That Is All.

What I actually needed was this:

queued → completed

A segment is either queued (not done) or completed (done). There is nothing in between.

Here is how it works. The manifest lists every segment with a status. All segments start as queued. A worker picks up a queued segment, generates the audio, writes the output file to disk, and then — only after the file exists and is valid — marks the segment as completed in the manifest.

If the worker crashes mid-segment, the segment is still queued. The output file may exist partially on disk, but the manifest does not lie. The segment has not been marked completed. When the pipeline restarts, it scans the manifest, finds the segment still queued, and tries again. The partial file gets overwritten.

No stale detection. No retry counter. No failure state. The system always knows exactly what to do: process every queued segment until none remain.

The manifest is the source of truth. The output files are a consequence of the manifest, not the other way around.

Why This Works

This works because of a specific property of the workload: segment processing is idempotent. Generating audio for segment 47 produces the same output whether it is the first attempt or the fifth. There is no harm in reprocessing a segment that was partially completed by a crashed worker. The result is the same.

This is the key constraint. If processing were not idempotent — if re-running a segment produced different output each time, or if it had side effects — then two states would not be enough. You would need to track attempts and distinguish between "never tried" and "tried and failed."

But TTS generation is deterministic given the same input text and model settings. So idempotency holds, and two states are sufficient.

The Writing Pattern

The critical implementation detail is the order of operations:

  1. Generate the audio in memory
  2. Write the WAV file to disk
  3. Verify the file exists and is non-empty
  4. Update the manifest from queued to completed
  5. Flush the manifest to disk

The manifest update happens last, after the output is verified. If the process dies at any point before step 5, the segment remains queued. The output file might exist on disk from a partial run, but the manifest has not been updated, so the segment will be reprocessed.

Write state after work, never before. The system recovers by default, not by exception.

This is the same principle behind the pre-flight catalog pattern — write your intent before execution so recovery knows what should exist. But for resume logic specifically, the insight is the opposite direction: write completion after verification, so incomplete work is invisible to the recovery system.

When You Need More States

There are cases where two states are not enough. I have not encountered them in the TTS pipeline, but I can imagine them:

Non-idempotent processing. If each run produces different output (creative generation with temperature sampling), you need to distinguish "attempted" from "not attempted" to avoid wasting compute on acceptable-but-different reruns.

Destructive side effects. If processing a segment sends an email or charges a credit card, you absolutely need to track attempts. Reprocessing is not harmless.

Long-running segments with checkpoints. If a single segment takes thirty minutes and has internal progress, you might want checkpoint states within the segment. But at that point, your segment is too big — split it into smaller segments.

For the vast majority of batch processing pipelines — audio generation, file conversion, data transformation, report rendering — two states are enough. The work is idempotent. Reprocessing is cheap relative to the cost of getting it wrong.

The General Principle

The broader lesson is about minimal state machines in system design.

Every state you add to a system multiplies your testing surface. Two states means one transition to test. Five states means at least ten transitions, plus invalid transition handling, plus recovery from each state on crash. The combinatorics get bad fast.

The best state machine is the one with the fewest states that can still answer the question you actually need answered.

The question I needed answered was: "Is this segment done?" Not "When did it start?" Not "How many times has it been tried?" Not "Which worker is processing it?" Just: is it done?

Two states answer that question perfectly. Every segment that is queued needs work. Every segment that is completed does not. The pipeline picks up all queued segments on restart and runs until none remain.

Forty minutes of audio generation, eight hundred segments, and the recovery logic is a filter: segments.where(status == "queued"). That is the whole thing. No retries to configure, no stale timers to tune, no failure states to investigate.

The simplest design that works is not the simplest design you can think of. It is the simplest design that survives a crash and produces correct output. For batch processing pipelines, that is usually two states and a manifest write after verification. Try it before you build anything more complex.