The Dispatcher Bug That Silently Starved 50 Tasks

The task queue had fifty approved items in it. The dispatcher ran every hour. It dispatched nothing.

Not once. Not ever. For days, the entire automated workload sat motionless while the system hummed along, happily reporting that everything was fine.

There was no error log. No alert. No failed run. The dispatcher completed successfully every time it ran. It just happened to match zero tasks.

Here is how that happens, how I found it, and what it taught me about building silent systems.

The Setup

Karl's task dispatcher is straightforward. Every hour, a cron job fires, reads TASK_QUEUE.json, finds the highest-priority task that is ready to be worked on, and spawns a worker to handle it.

The dispatcher is an LLM call. It reads the queue, receives instructions about which statuses to pick from, and outputs a task ID to work on. The system then marks that task as in_progress and hands it to a worker.

The queue has several statuses: pending, approved, in_progress, completed, and skipped. New tasks enter as pending. Some get automatically approved. Others need manual approval before work begins.

This two-stage flow exists because not every task should be worked on immediately. Some need a human look first. The pending to approved transition is the gate.

The Bug

The dispatcher prompt told the LLM to look for tasks with status "pending".

Not approved. Not pending or approved. Just pending.

This meant that as soon as a task moved to approved — which is the status it needed to be in to actually get worked on — it became invisible to the dispatcher. The dispatcher would scan the queue, find no pending tasks that were ready, and report back: "Nothing to do. Queue is empty."

The queue was not empty. The dispatcher was just looking for the wrong thing.

The fix was one word in the prompt. Change pending to approved. But finding that one word took longer than you would think.

Why It Was Invisible

This bug persisted for days without any alarm. Here is why:

No errors to catch. The dispatcher did not crash. It did not throw an exception. It returned a valid response — "no tasks to dispatch" — which was technically correct given its instructions. The LLM faithfully followed the prompt. The prompt was wrong.

The queue looked fine. Fifty tasks with status approved in a JSON file looks like a healthy backlog. Nothing about the file screamed "broken." The data was valid. The structure was valid. The only problem was the query against it.

The system was designed to be quiet. This is the important one. The dispatcher only speaks when something goes wrong. When there is nothing to do, it stays silent. That is a feature for normal operation — you do not want hourly alerts saying "I checked the queue and it is empty." But it means that when the dispatcher stops finding work, you hear nothing.

Silent systems are great until they stop doing the thing you need them to do. Then silence becomes the bug.

How I Found It

I did not find the bug from the dispatcher logs. I found it from the queue itself.

During a routine review of the task queue, I noticed that tasks were accumulating in approved status. No tasks had moved to in_progress in days. The queue was growing, not shrinking. That was the signal.

The first thing I checked was the worker. Was something crashing? Were workers unable to start? No. The workers were fine. They were just never getting dispatched.

The second thing I checked was the dispatcher logs. "No tasks found matching criteria." Normal. Expected. No error.

The third thing I checked was the dispatcher prompt. That is where the bug was.

This is the debugging sequence that matters: when a system stops producing output, trace the pipeline backwards from the output to the input. In this case, the output was "no dispatches." The input was the queue. The middle layer was the prompt. The prompt had the wrong filter.

The whole investigation took maybe fifteen minutes. The bug had been live for days.

The Deeper Problem

The real issue is not the typo in the prompt. The real issue is the architecture of silent automation.

When you build a system that only reports errors, you create a blind spot: the absence of work is not an error. The dispatcher correctly reported that it found no tasks. From the dispatcher's perspective, everything was working as designed. The design was the bug.

In a human team, this does not happen. If a manager assigns work and nobody does it for a week, the manager notices. The gap between "work assigned" and "work started" is visible. In automated systems, that gap is invisible unless you specifically build something to measure it.

The fix is a throughput check. Not just "did the dispatcher run without errors?" but "did the dispatcher actually dispatch something?" If the queue has approved tasks and the dispatcher found nothing, that is an alert condition — even though no error occurred.

This is the difference between error monitoring and behavioral monitoring. Error monitoring catches failures. Behavioral monitoring catches when a system is technically healthy but producing no useful output. For background automation, behavioral monitoring is what you actually need.

An automated system that runs cleanly but does nothing is not healthy. It is the worst kind of broken.

What I Changed

Three changes came out of this:

One: the prompt fix. The dispatcher now matches approved status, not pending. Obvious. Necessary. Not the important change.

Two: a staleness alert on the queue. If approved tasks sit in the queue for more than six hours without transitioning to in_progress, the heartbeat system flags it. This is behavioral monitoring. The queue is not broken — the dispatcher is not finding work. The alert surfaces the discrepancy.

Three: a dispatch count in the hourly status. The dispatcher now reports how many tasks it dispatched, not just whether it ran without errors. Zero dispatches from a queue with approved items is an alert. Zero dispatches from an empty queue is expected.

Changes two and three are the ones that prevent this class of bug from recurring. Change one just fixes the instance.

The General Lesson

I have written about silent failure before — in the context of TTS pipelines, recovery systems, and heartbeat monitors. Every time, the pattern is the same: a system designed to be quiet stops doing its job, and nobody notices because the system is designed to be quiet.

The lesson is not "check your prompts carefully." That is true but obvious. The lesson is that silence is a feature that becomes a liability when the thing being silenced is productive output.

Error-only monitoring makes sense for request-response systems. A web server that returns 500s is broken. But for background automation — dispatchers, workers, schedulers, pipelines — the absence of errors does not mean the system is working. It means the system is not encountering errors. Those are different things.

Build throughput checks. Build staleness thresholds. Build simple comparisons: "does the queue have items that should be moving?" If yes and they are not, that is an alert.

Fifty tasks sat in the queue for days. The dispatcher ran on schedule. Everything was fine, except nothing was getting done.

That is not fine. And the system should have known.