When Your Multi-Worker AI Pipeline Starts Fighting Itself

The TTS pipeline was running smoothly. Two workers processing audio segments in parallel, each one loading the Qwen3-TTS model into its own VRAM allocation on a shared RTX 4070 Ti. The heartbeat monitor showed green. The task queue was draining. Everything looked fine.

Then the render time for a full chapter jumped from 40 minutes to two hours.

No errors. No crashes. No warning messages. The workers were still responding to heartbeat polls. The task queue was still advancing. The system was, by every metric I monitored, operational. It was just taking three times as long as it should have.

The culprit was not a bug in the TTS model. It was not a corrupted input file. It was not even a performance regression. It was something far dumber: two workers had been assigned the same ID, and nobody was checking.

Building planning-first AI systems, one pipeline at a time. Get weekly deep dives into local AI, Rust, and agent orchestration — no fluff, no hype.

What Happened

My narrator-tts pipeline spawns worker processes from a pool. Each worker gets an ID — an integer assigned at spawn time based on the number of currently active workers. Worker 1, Worker 2, Worker 3. When a worker finishes its batch and exits, the next spawn reuses the lowest available ID.

The bug was in the spawn logic. Under certain timing conditions — specifically, when a worker exited between the availability check and the actual process creation — the dispatcher could assign an ID that was still held by a running worker.

The condition was rare enough that it never triggered during testing. It required a specific interleaving of heartbeat timing, task completion, and spawn scheduling. But in a long-running overnight batch processing 400+ audio segments, the window eventually opened. Worker 2 exited. The dispatcher calculated the next available ID as 2. Before the spawn completed, the scheduler reassigned ID 2 to a new worker. Now there were two Worker 2s.

Neither worker knew about the other. Both loaded the full TTS model into VRAM. Both started pulling tasks from the queue.

Two processes with the same identity, sharing the same GPU, pulling from the same queue. The system had no mechanism to detect this because, as far as it knew, there was only one Worker 2.

The VRAM Squeeze

An RTX 4070 Ti has 12.3 GB of VRAM. A single Qwen3-TTS worker, with model weights, KV cache, and audio processing buffers, needs about 9.4 GB. Two workers fit — barely — with 2.9 GB of headroom for the OS and other GPU tasks.

Four workers do not fit. Four was never the intention. But that is what happened: Worker 1, Worker 2 (the original), Worker 2 (the duplicate), and Worker 3. The combined VRAM allocation hit 19.5 GB on a 12.3 GB card.

The GPU did what GPUs do when they run out of VRAM: it started paging to system RAM. CUDA memory allocations fell back to unified memory, which transparently swaps GPU memory pages to CPU memory over the PCIe bus. No error, no crash, no warning. Just a 10x slowdown in every kernel launch.

Each TTS inference that should have taken 2 seconds now took 20. The workers were technically still producing audio. It was correct audio. It just arrived at a pace that made the pipeline useless.

Why Nothing Caught It

The monitoring system was designed to detect specific failure modes:

  • Worker crash — heartbeat goes stale, dispatcher respawns. Works perfectly.
  • Task timeout — a segment exceeds its processing budget, worker is killed. Works perfectly.
  • VRAM exhaustion — the auto-reserve system refuses to spawn new workers when VRAM is low. Works perfectly, but only checks before spawning, not after.

None of these detected the duplicate because the duplicate was not a crash, a timeout, or an over-allocation. It was a logic error in process identity. The monitoring system trusted that worker IDs were unique because the spawn logic was supposed to guarantee uniqueness. When that assumption broke, every downstream system continued operating on false data.

The heartbeat monitor showed Worker 2 alive — correct, since at least one Worker 2 was alive. The task queue showed segments being completed — correct, since both duplicates were processing work. The VRAM monitor showed 11.7 GB used, which was high but within the expected range for two workers. Except it was not two workers. It was four, and 7.8 GB had been silently paged to system RAM.

The Fix

The solution has two parts, and the second one matters more than the first.

Part 1: Fix the spawn race. The availability check and the ID assignment are now atomic. A single lock guards the ID allocation, and the lock is not released until the worker process has confirmed its identity via the IPC handshake. No more duplicate assignments.

Part 2: Detect duplicates anyway. Even with the race fixed, I do not trust the spawn logic to be perfect forever. The dispatcher now maintains a registry of active workers keyed by process PID, not just by logical ID. Every heartbeat includes the PID. If two heartbeats arrive with the same worker ID but different PIDs, the dispatcher kills the younger process (lower start timestamp) and logs an alert.

fn handle_heartbeat(&mut self, worker_id: u32, pid: u32, ts: Instant) {
    if let Some(existing) = self.workers.get(&worker_id) {
        if existing.pid != pid {
            // Duplicate detected — kill the newer process
            self.kill_duplicate(worker_id, pid, ts);
            self.alert_system.notify(
                format!("Duplicate worker {} detected: PID {} vs PID {}",
                    worker_id, existing.pid, pid)
            );
        }
    }
}

The detection runs on every heartbeat — every 30 seconds. If a duplicate somehow slips through again, it will survive for at most 30 seconds before being killed.

The General Principle

The deeper lesson here is about identity assumptions in distributed systems — even when "distributed" just means two processes on the same machine.

Every monitoring system, every health check, every alerting rule is built on assumptions about what the system's state can be. When those assumptions hold, the monitoring works. When they break — when a worker ID is not unique, when a task is in a state the state machine does not model, when a file exists but is empty — the monitoring happily reports green while the system degrades.

Monitoring does not protect you from states you did not think to check for.

The fix is not to add more monitors. It is to treat identity assertions as runtime invariants, not compile-time assumptions. At every boundary where components communicate — every heartbeat, every IPC message, every task claim — verify that the assumptions you built the system on still hold. Not because you expect them to fail, but because the cost of a duplicate worker is hours of silent degradation and the cost of a PID check is one integer comparison.

The overnight batch now finishes in 40 minutes. The duplicate detection has fired exactly once in the three weeks since I deployed it — a race I had not considered involving a worker that exited during GPU initialization. It killed the duplicate, logged the alert, and the batch continued without intervention.

That is the goal. Not a system that never fails, but a system that notices when it does.

The memory crisis that exposed this bug is covered from the system administration angle in 32 GB RAM Is Not Enough: A Memory Crisis Postmortem. That post is about what was eating the memory and the recovery; this one is about the duplicate worker detection system built afterward.