What 685 Tests Taught Me About Testing Rust AI Pipelines

The narrator-tts workspace has 685 tests as of the last commit. That number sounds impressive until you realize that about 500 of them run in under a second, and the remaining 185 take anywhere from thirty seconds to three minutes because they need a GPU.

The split is not accidental. It is the result of months of moving tests around, drawing boundaries, and learning the hard way what happens when you mix pure logic tests with GPU-dependent integration tests. Here is what the journey taught me.

The Boundary That Matters Most

Building planning-first AI systems, one pipeline at a time. Get weekly deep dives into local AI, Rust, and agent orchestration — no fluff, no hype.

The single most important decision in the test suite was drawing a line between tests that need hardware and tests that do not.

In the narrator-tts workspace, the qbench core library holds all the domain logic — audio chunking, QA scoring, silence detection, manifest serialization. None of it touches CUDA. None of it loads a model. None of it allocates VRAM. The library has over 500 tests that run in milliseconds because they test logic, not inference.

Then there are the integration tests. These live in a separate tests/ directory and require the backend-tch feature flag. They load actual models, run actual inference, and verify actual audio output against expected spectrograms. They are slow, they are hardware-dependent, and they are essential.

The boundary is not "unit tests vs integration tests." The boundary is "can this test run on a machine without a GPU?" If the answer is yes, it belongs in the fast suite. If the answer is no, it belongs behind a feature flag.

This separation has a concrete payoff. When I run cargo test --lib, I get 500 results in under five seconds. When I run cargo test --test integration --features backend-tch, I get the remaining results in three minutes. I choose which suite to run based on what I changed. Most of the time, the fast suite is enough.

Mocking CUDA Without Mocking CUDA

Here is a secret: I do not mock CUDA. I never have. Instead, I structure the code so that CUDA is never in the path for unit tests.

The qbench library defines a ModelBackend trait with methods like infer, load_model, and get_vram_usage. The real implementation — the one that wraps libtorch and manages CUDA contexts — lives in the backend-tch feature module. Unit tests use a MockBackend struct that implements the same trait but returns canned data from a Vec.

pub trait ModelBackend {
    fn infer(&self, input: &[f32]) -> Vec<f32>;
    fn model_name(&self) -> &str;
    fn vram_usage(&self) -> usize;
}

struct MockBackend {
    outputs: Vec<Vec<f32>>,
    call_index: usize,
}

impl ModelBackend for MockBackend {
    fn infer(&self, input: &[f32]) -> Vec<f32> {
        let result = self.outputs[self.call_index.min(self.outputs.len() - 1)];
        self.call_index += 1;
        result
    }
}

The tests verify that the chunking logic, QA scoring, and pipeline orchestration behave correctly regardless of what the backend does. The mock backend returns predictable data. The tests check the logic around the model, not the model itself.

The model inference is tested in integration tests against the real GPU. That is the only place where CUDA correctness matters. Everywhere else, I am testing what the code does with the output, not whether the output is correct.

Mocking frameworks are overrated. A trait and a struct with hardcoded return values will cover 90% of what you need. The other 10% probably should not be mocked at all.

The Four Timeouts You Can Ignore

Over months of running the test suite, I identified four tests that intermittently time out. Every single one is an environmental issue, not a code bug. Chasing these false failures wasted hours before I learned to recognize them.

VRAM contention timeout. When the integration tests run while another process is using the GPU — a browser hardware acceleration, a model left loaded in a previous process — the test takes longer because CUDA context initialization stalls. The test itself is fine. Killing the other process fixes it.

Model download timeout. The integration tests load model files from disk. When the models are in the system file cache, this is instant. When they have been evicted from cache — usually because I was doing something else with my RAM — the disk read adds seconds. This is filesystem behavior, not code behavior.

Audio device initialization timeout. One test verifies that the output WAV file can be opened by the OS audio stack. On Windows, the audio device API sometimes takes two seconds to respond if nothing has touched it recently. The test is checking a property of the OS, not the code.

Thread pool warmup timeout. The multi-worker integration test spawns a process pool. The first spawn of the day takes longer because the OS is loading the worker binary from disk. After the first run, subsequent spawns are fast because the binary is cached.

If a test times out, check whether something else is competing for the resource before you start debugging the test. Nine times out of ten, the test is telling you about the environment, not about the code.

I added a #[ignore] attribute to the audio device test because its failure mode has nothing to do with the pipeline. The other three I handle with generous timeout values and a comment explaining the environmental cause. Anyone reading the test knows why it might be slow.

Test What Changes, Not What Works

The 685 tests are not evenly distributed across the codebase. The chunking module has 140 tests. The QA scoring module has 90. The manifest serializer has 60. The CUDA backend has 12.

That distribution is not an accident. It reflects where bugs actually happen.

Chunking logic is where I have had the most bugs. Audio segments split at the wrong boundaries, silence detection thresholds that work for one narrator but not another, edge cases with very short or very long segments. Every bug I fix in chunking gets a regression test. The test count grows because that is where the complexity lives.

The CUDA backend has 12 tests because once it works, it tends to keep working. The model loads, produces output, and the output is either correct or it is not. There is not much logic to test — it is a thin wrapper around libtorch function calls.

Write tests where the bugs are, not where you think bugs might be. If a module has 100 tests and never fails, you are over-testing it. If a module has 5 tests and fails weekly, you are under-testing it. The test suite should concentrate where the complexity concentrates.

What I Would Tell Myself Six Months Ago

If I could go back to when the test suite had 30 tests and everything was in one file, I would say three things.

Draw the GPU boundary on day one. Even if you only have three tests, put the GPU-dependent ones behind a feature flag from the start. Moving them later is tedious and error-prone.

Do not mock the GPU. Mock the interface above it. If you are mocking CUDA calls, you are mocking the wrong thing. Define a trait, implement it twice — once for real, once for tests — and never let CUDA appear in your unit tests.

Ignore environmental flakes, document why. A test that fails because the OS audio device was sleepy is not a bug. Tag it, document it, and move on. Your future self will thank you for the comment.

685 tests is a lot. But the number that matters is not the count. It is how fast the fast ones run, and how reliable the slow ones are when you need them.