32 GB RAM Is Not Enough: A Memory Crisis Postmortem

The audiobook pipeline had been running for three hours. Forty-seven chapters generated, twenty-three remaining, and the progress bar was moving steadily. Then it stopped.

No error message. No crash. The worker processes just... halted. One was waiting on a GPU allocation that would never come. The other had been killed by the OS but nobody told the orchestrator.

When I opened Task Manager, the numbers looked wrong. Very wrong.

270 MB of RAM free. 6.2 GB paged to disk. The machine had 32 GB installed, and it was effectively out.

The Timeline

Here is what happened, reconstructed from logs and process snapshots:

Hour 0: Pipeline starts. Two qbench-worker processes spawn, each loading the TTS model into VRAM (~6 GB each). System RAM sits at ~14 GB used. Everything looks fine.

Hour 1: Chrome is open with 14 tabs — documentation, a YouTube video I forgot to close, two ChatGPT windows, a Google Doc. RAM climbs to 18 GB. Still healthy.

Hour 2: Steam updates download in the background. Spotify starts playing a playlist. Grammarly has loaded its background service. The ChatGPT desktop app refreshes two conversations. RAM hits 24 GB. The system starts paging.

Hour 2, Minute 40: The orchestrator's heartbeat detects that worker-1 hasn't produced output in 90 seconds. It assumes the worker died and spawns a replacement. But worker-1 isn't dead — it's just starved. The replacement loads the TTS model into VRAM, pushing total VRAM usage past the card's limit.

Hour 3: Worker-1 gets OOM-killed by the OS. Worker-2 and the new worker-3 are both alive but thrashing against the pagefile. VRAM is oversubscribed. Everything stalls.

The pipeline didn't fail because of a bug in the TTS engine. It failed because the machine ran out of memory and nobody noticed until it was too late.

What Was Actually Eating RAM

After killing everything and starting fresh, I measured each component:

Two TTS workers added 2.4 GB on top of that. Windows alone was consuming 4.5 GB. The machine had maybe 18 GB of headroom for actual work — and the pipeline needed closer to 20 when you counted model loading, audio buffering, and the orchestrator itself.

A 32 GB machine is not a 32 GB machine. It is a 32 GB machine minus everything you forgot was running.

The real bottleneck wasn't any single process. It was the accumulation of small consumers that individually looked harmless but collectively consumed a third of available RAM.

The Duplicate Worker Problem

The orchestrator's heartbeat recovery logic was designed to handle crashed workers. If a worker stops producing output for 90 seconds, assume it's dead and spawn a replacement.

This logic is correct when the worker has actually crashed. It is catastrophic when the worker is alive but starved.

The starved worker is still holding its VRAM allocation. The replacement worker tries to load the model again, doubling VRAM usage. Now both workers are competing for resources, and neither can make progress. The heartbeat timer fires again, and if you're unlucky, it spawns a third.

This is the same pattern that brought down the pipeline. The fix was straightforward: before spawning a replacement, check whether the original process is still alive. If it is, don't spawn a duplicate. Kill the original, wait for its VRAM to free, then spawn fresh.

fn handle_stale_worker(&mut self, worker_id: &str) {
    if self.is_process_alive(worker_id) {
        // Worker is alive but stalled — kill it cleanly
        self.kill_worker(worker_id);
        self.wait_for_vram_release(worker_id);
    }
    // Now safe to spawn a replacement
    self.spawn_worker(worker_id);
}

A 15-line fix that would have prevented the entire incident.

The Pagefile Trap

Here is something I didn't fully appreciate until this incident: paging is not your friend when you're running GPU workloads.

When the OS pages memory to disk, it doesn't page the GPU's VRAM. VRAM is dedicated memory on the card. But the CPU-side processes that manage the pipeline — the orchestrator, the audio concatenation, the file I/O — those live in system RAM. When that RAM gets paged, those processes slow to a crawl.

The pipeline wasn't GPU-bound in those final minutes. It was disk-bound, because Windows was frantically swapping pages in and out to keep everything alive. The GPU was sitting there with spare capacity, waiting for the CPU to hand it the next chunk of text.

The recovery playbook now includes a hard rule: if system RAM drops below 4 GB free, pause the pipeline. Don't try to push through. Stop, free memory, and resume.

The Recovery Playbook

After this incident, I put together a simple recovery procedure. Nothing fancy — just a checklist to run when the pipeline stalls and memory looks suspicious.

Step 1: Check Get-Process for duplicate workers. If two processes have the same worker ID, kill both. The orchestrator will respawn on the next heartbeat.

Step 2: Close everything that isn't the pipeline. Chrome, Spotify, Steam, the ChatGPT app — all of it. These are convenience tools, not infrastructure. They can reopen after the book is done.

Step 3: Check VRAM with nvidia-smi. If VRAM usage doesn't match the number of active workers, something didn't clean up. A dead worker can leave VRAM allocated if the process was killed forcefully.

Step 4: Flush and restart. The pipeline has a --flush-vram flag that terminates all workers, releases VRAM, and starts clean. Use it. Trying to recover in-place saves five minutes but risks another stall.

Step 5: Resume from the last completed segment. The segment catalog knows exactly where it stopped. Resume from there. Don't re-run completed chapters.

The whole recovery takes about three minutes. The first time it happened, it took forty minutes of poking around Task Manager and reading logs before I understood what had gone wrong.

What Changed

Three things changed after this incident:

The orchestrator now checks process liveness before spawning replacements. This is the fix that matters most. No more duplicate workers consuming VRAM for nothing.

The pipeline monitors system RAM and pauses when free RAM drops below 4 GB. A paused pipeline is recoverable. A crashed pipeline with corrupted state is a much bigger problem.

I close Chrome before starting a book-length run. This sounds stupidly simple, and it is. But 3.4 GB of RAM sitting in browser tabs that I'm not even looking at is 3.4 GB the pipeline can't use. The pipeline takes priority.

The Broader Lesson

This incident wasn't caused by a single failure. It was caused by accumulation — many small memory consumers that were each reasonable in isolation, but collectively pushed the system past its limit.

This is how most infrastructure failures happen in practice. It's rarely one thing. It's the compounding of several things that each seemed fine at the time.

When debugging resource exhaustion, don't look for the one big consumer. Look for the long tail of small ones.

32 GB felt like more than enough. It was more than enough for any single component. It was not enough for all of them at once, running simultaneously, with a duplicate worker that shouldn't have existed.

The fix wasn't buying more RAM — though that would help. The fix was understanding what was actually consuming memory, preventing the duplicate worker bug, and having a recovery playbook for when things go wrong anyway.

You don't prevent every crisis. You build systems that recover from them quickly and don't make them worse in the process.

The duplicate worker that triggered this crisis — and the VRAM fight it caused — is covered from the infrastructure angle in When Your Multi-Worker AI Pipeline Starts Fighting Itself. That post is about the detection system built afterward; this one is about what went wrong and why.