Using Local LLMs for Text Preprocessing at Temperature 0.3

The narrator-tts pipeline takes a manuscript and produces an audiobook. Between the raw text and the audio model sits a preprocessing layer that cleans, annotates, and structures the text. For months, this layer was a pile of regex rules and Python string operations. It worked, but it was brittle. Every new manuscript exposed a new edge case the regex did not handle.

The obvious fix was to use an LLM for preprocessing. The obvious problem with using an LLM for preprocessing is that LLMs are inconsistent. You ask them to normalize a contraction, and ninety-nine times out of a hundred they do it correctly. That hundredth time, they rewrite the entire sentence.

This is the story of how I built a three-stage text rewriter using local LLMs through Ollama, why temperature 0.3 turned out to be the only value that works, and why the post-processing layer matters as much as the LLM itself.

Why Local, Why Ollama

I run the entire TTS pipeline locally. The manuscripts are unpublished work that is not going through an API. Even if privacy were not a concern, latency would be — a 120,000-word novel generates roughly 2,000 preprocessing calls, and round-tripping each one through an API would take hours.

Ollama runs on the same machine as the TTS pipeline. A 1.7B parameter model loaded into VRAM responds in under 200ms per call. Two thousand calls at 200ms each is about seven minutes. That is acceptable for a pipeline that takes three hours to generate audio.

The model I use is Qwen2.5-1.5B-Instruct. It is small, fast, and surprisingly capable at constrained text transformation tasks. It is not writing poetry. It is normalizing punctuation, expanding abbreviations, and annotating speaker attribution. A small model handles these tasks fine — and it handles them much faster than a 70B model would.

The Three Stages

The rewriter pipeline has three stages, each a separate LLM call with a different prompt. Running them as separate calls instead of one big prompt was an early design decision that paid off immediately.

Stage 1: Text Normalization

The first stage takes raw manuscript text and produces clean, normalized text. Abbreviations get expanded ("Dr." becomes "Doctor"), numbers get written out ("42" becomes "forty-two"), em-dashes get consistent spacing, and stray Unicode characters get replaced with their ASCII equivalents where possible.

Input:

Dr. Elise checked the read-out — 42.7°C was 3° above normal.

Output:

Doctor Elise checked the read-out — forty-two point seven degrees Celsius was three degrees above normal.

The prompt is specific about what to normalize and what to leave alone. It includes a list of abbreviations to expand, a directive to preserve dialogue exactly, and a directive to never reorder or rephrase the text.

The normalization prompt's most important instruction is: "Output the full text. Do not summarize, do not abbreviate, do not skip any words." Without this, the model occasionally decides to shorten long paragraphs.

Stage 2: Dialogue Annotation

The second stage takes the normalized text and annotates dialogue with speaker information. This is where the LLM does something that regex cannot — it reads context to determine who is speaking when there is no explicit dialogue tag.

Input:

"Are you serious?" Elise asked.

"Dead serious."

Output:

[spk=Elise]"Are you serious?"[/spk] Elise asked.

[spk=Marcus]"Dead serious."[/spk]

The prompt provides the character list for the scene, the preceding context, and explicit instructions to use only names from the character list. This stage connects to the dialogue tag parsing work I wrote about previously — the LLM handles the ambiguous 15% that the structural parser and turn-taking logic could not resolve.

Stage 3: Reading Annotation

The third stage adds reading directives — pause markers, emphasis hints, and pacing notes. Not for every sentence, but for moments where the text structure implies a specific reading that a TTS model might miss.

Input:

He stared at the door. Three seconds. Four. Then he turned the handle.

Output:

He stared at the door. [pause=500]Three seconds.[pause=250] Four.[pause=500] Then he turned the handle.

The LLM identifies moments where punctuation does not adequately convey the intended rhythm. Short sentences after long ones. Fragmented text. Dramatic pauses implied by paragraph breaks. These annotations give the TTS model explicit timing instructions instead of relying on its built-in heuristics.

Why 0.3

Temperature is the single most important parameter in this pipeline. I tested every value from 0.0 to 1.0 in increments of 0.1, running the same 50-paragraph test manuscript through each stage at each temperature.

At temperature 0.0, the model produced identical output for identical inputs — which sounds ideal until you realize it also produced identical mistakes. When the model was wrong about a speaker attribution, it was consistently wrong across every run. There was no variation to average out.

At temperature 0.5 and above, the model started getting creative. Stage 1 would occasionally rephrase sentences instead of normalizing them. Stage 2 would invent character names that did not exist in the character list. Stage 3 would add dramatic pauses to mundane sentences.

At temperature 0.3, the model followed instructions consistently without being locked into a single failure mode. When it was unsure about a speaker attribution, it picked the most likely candidate from the character list instead of inventing a name. When it normalized text, it stayed close to the original. The variations between runs were small enough that they did not affect the final audio quality.

Temperature 0.3 is the point where the model has enough flexibility to avoid systematic errors but not enough freedom to hallucinate. For text transformation tasks, this is the sweet spot. I have not found a reason to change it.

The Post-Processing Layer

The LLM produces good output most of the time. "Most of the time" is not good enough for a production pipeline. The post-processing layer is what makes the system reliable.

Length verification. The normalized output should be approximately the same length as the input — within 20%. If stage 1 output is significantly shorter, the model probably summarized instead of normalizing. The post-processor detects this and falls back to the original text.

Character list validation. Every speaker tag in stage 2's output must match a name in the provided character list. Unknown names get replaced with the last known speaker. This catches the model's occasional tendency to use a character's first name when the list has their full name, or vice versa.

Annotation format validation. Stage 3's pause markers must match the expected format and have reasonable durations. A [pause=50000] is clearly a hallucination. The post-processor caps pause durations at 2000ms and strips malformed annotations.

Diff-based review. For stage 1, the post-processor runs a word-level diff between input and output. If more than 10% of words changed — excluding the expected normalizations like abbreviation expansions — the segment is flagged for manual review. This catches the cases where the model decides to be an editor instead of a normalizer.

The LLM is one component of the pipeline, not the pipeline itself. Treat its output as untrusted. Verify everything. The post-processing layer is where the reliability comes from, not from the model.

What I Would Tell Someone Building This

Run stages separately. I tried a single-prompt approach first — normalize, annotate speakers, and add reading directives in one call. The model did all three tasks poorly because the prompt was too complex. Splitting into three calls with focused prompts improved quality dramatically, and the total processing time only increased by about two minutes across a full book.

Use the smallest model that works. I tested with a 7B model first. It was marginally more accurate on dialogue attribution and noticeably slower. The 1.5B model at temperature 0.3 with post-processing validation produces results that are just as reliable for this task. Save the big models for tasks that need reasoning, not text transformation.

Build the post-processor first. Before writing a single LLM prompt, write the validation layer. Define what correct output looks like. Define what failure looks like. Then build the prompt to produce output that passes validation. If you build the LLM call first, you will spend weeks trying to get the model to be reliable enough without a safety net.

The LLM is a tool in the pipeline. It is not the pipeline. The pipeline is the prompts, the post-processing, the fallbacks, and the validation rules that together produce consistent output from an inherently inconsistent system.

The pipeline described here was built to produce the audiobook for The Incubator. Read more about the Etrath series.