People don't speak in order

Here is a sentence I could easily say to my phone on a Tuesday night:

“Okay, remind me to email Mr. Lee on Friday, actually Thursday, about the stats thing, and before that I need to finish the slides. Oh, and when I send it, cc Nik.”

The likely reading is one email, due Thursday, with the slides finished first and Nik on the cc line. The listener has to connect those pieces and treat Thursday as a correction.

Now look at what the transcript actually contains. The word “Friday” is in there. The email task is split across three places. The slides come second in the sentence but first in time. “It” in the last clause points back to an email that was mentioned two clauses ago. If you hand that string to a system that just reads left to right, every one of those is a chance to get it wrong.

I’ve been thinking about this since building DFM Voice at the VoiceOS hackathon, and I keep running into the same problem: a transcript alone is not enough state for a voice assistant that takes actions. It records what was said, in the order it was said. What you actually want is what the person means right now, and that is a different object.

Speech is not a draft you can edit

When you type, you fix your mistakes before anyone sees them. The backspace key hides the repair. When you talk, the repair is part of the output. You say the wrong thing, you notice, and you fix it out loud, and the listener has to work out what the final version is.

Linguists have studied this for a long time. Willem Levelt’s 1983 paper “Monitoring and self-repair in speech” looked at 959 spontaneous self-repairs and described repair as happening in three phases: you monitor your speech and interrupt yourself when you catch a problem, there is a hesitation (often with an editing term like “uh”), and then you produce the repair.¹ One of the findings I like most is that the editing term plus the first word of the repair usually gives the listener enough to connect the fix back to the original.¹ So in “Friday, actually Thursday”, the word “actually” plus “Thursday” is the listener’s cue that “Thursday” replaces “Friday” and not, say, adds a second reminder.

Elizabeth Shriberg’s 1994 dissertation gave the structure names that speech researchers still use. A disfluency has a reparandum (the part that gets replaced), an interruption point, an optional interregnum (the “uh” or “I mean” or “actually”), and the repair.² Mapped onto my sentence:

remind me Friday   |   actually   |   Thursday
  reparandum      IP  interregnum      repair

This is not some rare edge case. In a later paper Shriberg reports that disfluencies affect about ten percent of words and up to a third of utterances in natural conversation.³ Those rates depend on the corpus and what counts as a disfluency, but they are high enough that repairs deserve ordinary handling.

Even the filler words carry information. Herbert Clark and Jean Fox Tree argued in 2002 that “uh” and “um” are actual English words that speakers plan and produce, and that they signal an upcoming delay: “uh” for a minor one, “um” for a major one.⁴ That claim has been debated, and some later work disputed whether fillers really signal delay this cleanly,⁴ so I wouldn’t build a system that leans hard on it. But the general point stands. Spontaneous speech is full of signals about the speaker’s own process, and a cleaned-up transcript throws a lot of them away.

Not just repairs

Self-repair is the cleanest example, but it is only one of the ways speech comes out of order. The ones I keep running into:

• Tasks out of order. “Before that, finish the slides” is said after the email but happens before it. Word order and time order disagree.
• Pronouns and references. “When I send it” only means something if you know what “it” is. “The stats thing” assumes shared context.
• Self-interruptions. People start one thought, abandon it, and come back to it later. “Remind me to… oh wait, first, can you check if… no, never mind, remind me to call Mom.”
• Late attachments. “Oh, and cc Nik” modifies something that was already “finished” from the speaker’s point of view a sentence ago.

The common thread is that later words change the meaning of earlier ones. The transcript is append only. Meaning isn’t.

Why next-token prediction over a transcript loses track

The obvious way to build a voice assistant today is to run speech recognition, append the text to a running conversation, and let a language model read the whole thing every turn and decide what to do. Honestly, this works surprisingly well for short exchanges. Models are good at resolving “Friday, actually Thursday” when it’s right there in one sentence.

The problem shows up over time. The model has to re-derive the current state of every task from the full history on every turn, and that history includes all the wrong versions. The Friday is still in there. So is the abandoned thought. Every turn is another chance to resurrect one of them.

There is decent evidence that this kind of re-derivation gets worse as conversations go on. Laban and colleagues ran more than 200,000 simulated conversations and found that LLM performance dropped a lot when instructions were spread across multiple turns instead of given all at once, mostly because the models got less reliable, not because they got less capable.⁵ Their read is that models make assumptions early, commit to an answer too soon, and then lean on it, so once a conversation takes a wrong turn the model tends not to recover.⁵ That is exactly the failure I’m worried about. A wrong early guess about which day the reminder was for is hard to shake if the model keeps rereading its own guess.

Separately, Liu et al. showed that models are better at using information at the start or end of a long context than information in the middle, with a U-shaped accuracy curve as the relevant passage moves through the input.⁶ A correction made ten minutes into a conversation can easily end up in the middle.

None of this means the model is bad. It means we are asking it to do two jobs at once: understand what the person just said, and also act as the database of everything that is currently true. I think those should be separate.

Turn speech into state changes

The alternative I would try is to stop treating speech as text to accumulate and start treating each utterance as a set of state changes. Each thing the person says gets parsed into small, explicit events. The events go into a log. The current state of every task is computed from the log. The model reads the state, not the raw history.

This is basically event sourcing. Martin Fowler’s 2005 article defines it as capturing all changes to application state as a sequence of events, and points out what you get from that: you can rebuild the state from scratch by replaying the events, you can ask what the state looked like at any earlier point, and you can fix a past mistake by reversing a bad event and replaying.⁷ All three of those turn out to matter a lot for voice.

Here is what my example sentence might turn into:

[
  {
    "event": "create_task",
    "task_id": "t1",
    "fields": {
      "action": "email",
      "recipient": "Mr. Lee",
      "topic": "stats",
      "due": "Friday"
    },
    "source": {
      "utterance_id": "u17",
      "span": "remind me to email Mr. Lee on Friday"
    },
    "timestamp": "2026-09-15T21:14:02-07:00",
    "confidence": 0.92
  },
  {
    "event": "modify_task",
    "task_id": "t1",
    "field": "due",
    "from": "Friday",
    "to": "Thursday",
    "reason": "self_repair",
    "source": { "utterance_id": "u17", "span": "actually Thursday" },
    "timestamp": "2026-09-15T21:14:03-07:00",
    "confidence": 0.95
  },
  {
    "event": "create_task",
    "task_id": "t2",
    "fields": { "action": "finish", "object": "slides" },
    "source": {
      "utterance_id": "u17",
      "span": "before that I need to finish the slides"
    },
    "timestamp": "2026-09-15T21:14:05-07:00",
    "confidence": 0.88
  },
  {
    "event": "modify_task",
    "task_id": "t1",
    "field": "cc",
    "to": ["Nik"],
    "source": { "utterance_id": "u17", "span": "when I send it, cc Nik" },
    "timestamp": "2026-09-15T21:14:08-07:00",
    "confidence": 0.81
  }
]

A few rules make this work.

Every event has a source. It points back to the exact span of speech that produced it. If the assistant gets something wrong, you can see which words caused it. When a user says “no, I didn’t say that”, you know which event to look at.

Every event has a timestamp and a confidence. Confidence lets the system decide whether to act or ask. Timestamps let you replay, which is the “what did the state look like before the correction” query from Fowler’s list.

Never silently overwrite. The Friday doesn’t disappear. A modify_task event records that it changed from Friday to Thursday and why. That sounds like bookkeeping, but it’s what makes undo possible. If the recognizer misheard “actually” and the person really did want two reminders, you can undo exactly one event instead of guessing what the old state was. And if the user asks “wait, what did I say?”, the answer is in the log.

Incremental updates instead of regenerating everything

There is also a cost argument. If every turn means the model rereads the whole transcript and regenerates its picture of the world, the token bill grows with the length of the conversation, every single turn. With an event log, each new utterance only has to produce the events for what changed. The model’s prompt can be the current task state (small, and bounded by how many tasks exist) plus the latest utterance.

The dialogue research community landed on a similar idea a while ago. SOM-DST, from Kim and colleagues at ACL 2020, treats the dialogue state as a fixed-size memory and first predicts an operation for each slot, then only generates new values for the slots that actually need updating, instead of regenerating the full state every turn.⁸ Different architecture, same instinct: most turns change a small part of the state, so only touch that part.

The task object

The log is the source of truth, but you need a current view to plan from. Here is roughly what I think a canonical task should look like after replaying the events above:

{
  "task_id": "t1",
  "action": "email",
  "recipient": "Mr. Lee",
  "topic": "stats",
  "due": "2026-09-17",
  "cc": ["Nik"],
  "status": "proposed",
  "confidence": {
    "semantic": 0.9,
    "temporal": 0.95,
    "dependency": 0.78
  },
  "relations": [{ "type": "must_complete_before", "from": "t2", "to": "t1" }],
  "history": ["e101", "e102", "e104"]
}

Factorized confidence

The part I care most about is that confidence is split into three numbers instead of one.

• Semantic: do I understand what the task is? Is it an email to Mr. Lee about stats?
• Temporal: do I know when? “Thursday” is clear. “Sometime next week” is not.
• Dependency: do I know how it relates to other tasks? Does “before that” attach to the email, or to something said earlier?

The reason to factor it is that each kind of uncertainty calls for a different question back to the user. If semantic confidence is low, you ask “who’s the email to?” If temporal is low, you ask “which Thursday?” A single blended score of 0.7 tells you something is off but not what to ask about, so the assistant either guesses or asks a vague “did you mean…?” that annoys everyone.

My view is that you should gate on the weakest dimension, not the average. A task you’re sure about except for the date is still a task you can’t schedule.

def needs_clarification(conf, bar=0.8):
    weakest = min(conf, key=conf.get)
    return (weakest if conf[weakest] < bar else None)

That returns which thing to ask about, which is more useful than a yes or no.

Relations between tasks

Tasks don’t live alone. I think three relation types cover most of what people say:

• must_complete_before: “before that, finish the slides.” The planner can’t mark the email ready until the slides are done.
• supersedes: “actually, forget the email, just text her.” The new task replaces the old one. The old one stays in the log, marked superseded, so it can come back if the user changes their mind again.
• requires_confirmation: anything with real side effects or low confidence. Sending a message on someone’s behalf is the obvious one.

[
  { "type": "supersedes", "from": "t3", "to": "t1", "source": "u21" },
  { "type": "requires_confirmation", "task": "t3", "why": "sends a message" }
]

This is where the graph part of “task graph” comes from. Once relations are explicit, ordering stops being a language problem and becomes a topological sort.

Four states

For this prototype, I would start with four states:

1. Proposed. The system heard something that sounds like a task. Nothing happens yet.
2. Confirmed. The user said yes, or every confidence dimension cleared the bar and the task doesn’t require confirmation.
3. Ready. Everything it depends on is done and there are no open questions. The planner can pick it up.
4. Verified. It was executed and the result was checked. Only now does the assistant say “done.”

The split between ready and verified matters more than it looks. A lot of assistants say “I’ve set a reminder” when what actually happened is they called an API and didn’t check the response. I would rather the system only claim things it has confirmed. Verification results get written back to the log as their own events, so a failure is just another state change the planner can react to.

The whole pipeline

Put together, it looks like this:

The language model still does a lot. Event extraction is a model call, and it’s the hard part: deciding that “actually” means replace, that “it” means the email, that “before that” is a dependency. But it does that job on one utterance at a time, with the current state as context, and its output is small and checkable. The planner works from a structured graph instead of from prose.

None of this is new to dialogue research, to be fair. Dialogue state tracking has been a named problem for over a decade. The second Dialog State Tracking Challenge in 2014 specifically added changing user goals to its setup, and found that trackers still degraded in mismatched conditions.⁹ MultiWOZ, released in 2018, is a set of about 10,000 human-to-human task-oriented dialogues with state labels, and became a standard benchmark for this.¹⁰ One detail I find kind of telling: when MultiWOZ 2.1 cleaned up the annotations, over 32% of state labels changed across 40% of turns.¹¹ Even the humans labeling the “correct” state had trouble keeping it straight.

The difference with LLM-era assistants is that the state isn’t a fixed set of slots like restaurant-area or hotel-stars anymore. People can ask for anything. So the state has to be open-ended, which is why I’d use a general task object with relations rather than a slot table. On the LLM side, MemGPT is a good example of the general move toward giving models explicitly managed memory instead of one long context, borrowing the idea of memory tiers from operating systems.¹² An event log is a more opinionated version of that idea for one specific domain.

Where this comes from for me

At the VoiceOS hackathon, we turned Dear Future Me from a website with forms into an iPhone app you just talk to. You tap a widget, it’s listening, and an agent decides what to keep and saves it in your words. VoiceOS handled the hard conversational parts, like knowing when you’ve finished talking.

What stuck with me from that build is how different the input becomes once you remove the form. A form forces people to put things in order. One field, one value. Take it away and you get speech the way it actually comes out, with all the restarts and corrections that Levelt and Shriberg describe. For a journaling app, a lot of that is fine, and honestly sounding like yourself is the point. But the moment an assistant has to do something with what you said, like schedule it or send it, the order problem stops being cosmetic.

I want to be clear that the event-sourced design in this post is my thinking about where to go next, not a description of how DFM Voice works today. It’s the architecture I’d want if I were building a voice assistant that takes actions.

What I don’t know yet

What I want is an assistant that lets someone work out what they mean while speaking. Changing your mind halfway through shouldn’t leave the software confidently following a plan you’ve already taken back.

The hard cases seem more interesting than perfectly tidy requests. I’d try unfinished thoughts, corrections, and sentences that could reasonably mean two different things. Sometimes the right response might be to update the plan; sometimes it might be to ask one short question. If the system could get better at making that choice, speaking to it could feel much less like filling out a form.

Sources