Teaching an AI to Read Dialogue Tags

The narrator-tts pipeline takes a manuscript and produces an audiobook. One of the features I wanted early on was distinct character voices — not just a different pitch for each speaker, but genuinely different vocal characteristics. The protagonist should not sound like the villain. The narrator should not sound like either of them.

Before you can assign voices, you have to answer a question that sounds trivial: who is speaking?

A human reader knows. They read the dialogue tag — "said Elise" — and they know. They infer from context when there is no tag. They track conversation turns. They recognize when a new paragraph means a new speaker.

A TTS pipeline does none of this automatically. It receives text. It produces audio. If you want different voices for different characters, the pipeline has to parse the manuscript, identify each speaker, and tag the text before sending it to the model.

This is a harder problem than I expected.

The Surface Problem: Quotes and Tags

The obvious starting point is dialogue tags. In a typical manuscript, spoken dialogue appears inside quotation marks, followed or preceded by an attribution:

"I'm not going back there," Elise said.

Simple enough. Find the text inside quotes, find the name in the tag, assign the voice. But manuscripts are not uniform. Over the course of a 120,000-word novel, every possible variation shows up.

Curly quotes. My manuscripts use Unicode curly quotes — " and " — not straight ASCII quotes. Some text editors convert automatically. Some do not. The parser has to handle both, and it has to handle the asymmetric opening and closing pairs.

Tags before the dialogue. Sometimes the attribution comes first:

Elise said, "I'm not going back there."

Tags in the middle. The speaker pauses, the narration describes an action, then the dialogue continues:

"I'm not going back," Elise said, crossing her arms. "Not after what happened."

This is one continuous speech by one character. The parser must recognize that both quoted sections belong to Elise, not two different speakers.

Tags after a beat of narration. Even trickier:

"I'm not going back." She crossed her arms and stared at the door. "Not after what happened."

No explicit tag at all on the second line. The reader knows it is still Elise because she is the focus of the narration and the dialogue continues her thought. The parser has to infer this.

The Deep Problem: Missing Tags

Roughly 30% of dialogue lines in a typical manuscript do not have an explicit tag. The author expects the reader to track the conversation through rhythm and context.

Consider this exchange:

"Are you serious?"
"Dead serious."
"You can't mean that."
"I do."

Four lines, no tags. A human reader who has been following the scene knows who is speaking — it alternates, following the established turn-taking pattern. But a parser seeing this in isolation has no idea.

The fundamental problem is that dialogue attribution is a semantic task, not a syntactic one. The speaker's identity is determined by context, narrative focus, conversation flow, and sometimes explicit tags. A regex cannot solve this.

This is where the parsing approach has to shift from pattern matching to something smarter.

The Three-Layer Approach

I ended up with a pipeline that handles dialogue attribution in three passes, each more sophisticated than the last.

Layer 1: Structural Parsing

The first pass handles the easy cases. Find all quoted text. Look for attribution patterns — said NAME, NAME said, NAME whispered, asked NAME, and the dozens of variations. Attach the quoted text to the identified speaker.

This layer also detects blockquotes (which in my manuscripts signal a system voice or ancient text), italics for emphasis, and plain narration paragraphs. It tags everything it can with high confidence and flags the rest for the next layer.

This layer catches about 60% of dialogue lines correctly. The ones with explicit, unambiguous tags.

Layer 2: Conversation Turn-Taking

The second pass handles untagged dialogue in the middle of a conversation. The rule is simple: in a two-person conversation with no explicit tag, speakers alternate. If line N is Elise, line N+1 is the other person, line N+2 is Elise again.

This works for most back-and-forth exchanges. It fails when a third character interjects, when someone has a long multi-paragraph speech, or when narration interrupts the rhythm. Those cases get flagged for layer three.

Adding turn-taking brings coverage up to about 85%. The remaining 15% are the genuinely ambiguous cases.

Layer 3: LLM Attribution

For the remaining untagged dialogue, I use a local LLM. The prompt gives the model the scene context — who is present, what the conversation is about, the preceding and following paragraphs — and asks it to identify the speaker.

The key insight here was using a low temperature. At temperature 0.3, the model is consistent across runs. It does not get creative with attributions. It looks at the context and picks the most likely speaker, which is usually correct.

An LLM for dialogue attribution is not a cop-out. It is the right tool for a semantic task that syntax alone cannot solve. The trick is constraining it — low temperature, structured output, and a verification pass that rejects implausible attributions.

The verification pass checks the LLM's output against a character list. If the model attributes dialogue to a character who is not in the scene, the attribution is rejected and the line defaults to the last known speaker. This catches hallucinations.

What Surprised Me

A few things I did not expect when I started this work:

Continuation speech is the most common edge case. A character speaks, the narration describes an action, the same character speaks again. Without recognizing this pattern, the parser assigns the second speech to a new speaker, which breaks the voice assignment for the audio. This happens constantly in real manuscripts.

Some authors never use "said." They use action beats exclusively — the character does something, and the dialogue follows with no tag. This is valid stylistically but murder for a parser. The action beat has to be parsed to identify the subject, and the subject has to be matched to a character name.

Internal monologue looks like dialogue. Thoughts in italics, sometimes in quotes, sometimes not. The parser has to decide: is this spoken aloud (needs a character voice) or internal (needs the narrator voice or the character's internal voice)? Context is the only signal, and context requires understanding the scene.

The Principle

The broader lesson is one I keep learning in different parts of the TTS pipeline: text is not data. A manuscript is a structured narrative document with implicit rules, conventions, and context that human readers process unconsciously. Asking a parser to extract structure from it is asking the parser to do something that is fundamentally a reading comprehension task, not a string processing task.

The three-layer approach works because it matches the difficulty of each case to the right tool. Structural parsing is fast and handles the majority of cases. Turn-taking logic handles conversations. An LLM handles the ambiguous remainder. Each layer only processes what the previous layer could not.

Build parsing in layers. Let each layer handle what it is good at, and pass the hard cases to something smarter. A single-approach parser — whether regex or LLM — will either miss cases or be too expensive to run on every line.

The result is not perfect. I estimate attribution accuracy at around 96% — which means roughly one in twenty-five dialogue lines gets the wrong voice. Most of the time, the error is assigning narration to the previous speaker or vice versa. It is noticeable if you are paying attention, but it does not ruin the listening experience.

Getting from 96% to 99% would require scene-level understanding — tracking who knows what, who is angry at whom, who is in the room. That is a different kind of problem. For now, 96% with a manual review pass is good enough to ship.