Where talking to my computer works better than typing

Twelve days ago I wrote about building DFM Voice at the VoiceOS hackathon. VoiceOS has stayed in my workflow since, and I have been trying to work out which parts of that experience might carry over to other voice tools.

Here is the claim. For getting a thought out of your head and into a computer, talking often feels easier to me than typing. For shaping that thought into something precise, typing still wins, and probably will for a while. Most of the interesting design work in the next few years is in the gap between those two.

Start with the numbers

The cleanest measurement I found is a Stanford study by Sherry Ruan, James Landay and colleagues. They had people enter short messages on an iPhone, either by typing on the keyboard or by speaking into a speech recognizer. In English, speech came in at 153 words per minute and the keyboard at 52. In Mandarin it was 123 against 43. Speech was close to three times faster in both languages.¹

The part I did not expect: the corrected error rate was lower for speech too, 5.30% against 11.22% for the keyboard. The uncorrected error rate was slightly higher for speech, 1.30% against 0.79%.¹ So speech was faster and made you fix fewer things along the way, but a few more mistakes slipped through to the end. Hold on to that last detail, because it comes back later.

That study was in a lab. For a bigger picture of how people actually type on phones, there is a study by Kseniia Palin and colleagues with about 37,000 volunteers. The average was 36.2 words per minute.²

These numbers are from different setups, so I would not read too much into the exact ratio. But the direction is not subtle. For those short-message tasks, speaking had a substantial speed advantage.

Why speed is not really the point

If speed were the whole story, voice would have won ten years ago. Speech recognition was already good in 2016. That year a Microsoft team reported a system with a 5.8% word error rate on the Switchboard conversational speech benchmark, against 5.9% for professional human transcribers.³ People called it human parity. That benchmark milestone did not, by itself, make dictation the default way people used computers.

The speed of the pipe matters less than what happens to your thoughts on the way through it.

When I type, I edit while I write. I start a sentence, delete half of it, rephrase it, check the spelling. By the time a thought hits the screen, it has been filtered. That is fine for an essay. It is bad for capture, which is the part where you are trying to get the raw idea down before it disappears.

I made this point in the DFM post about journaling: typing into a form makes you edit yourself, and talking does not.⁴ For a reflection notebook, sounding like yourself is the point. I now think this goes beyond journaling. A lot of what we do on computers starts with capture. A note for later, a bug I just noticed, a half-formed idea for a project, a message to a teammate. Typing puts a small editor between me and all of those.

Since the hackathon I have been talking my quick notes and ideas into the computer instead of typing them, and the main thing I noticed is that I capture more of them.

What actually changed

So if the recognition was good in 2016, why does voice feel different now? I think two things happened, and the second one matters more.

Recognition got robust, not just accurate

Benchmark accuracy and real-world accuracy are different things. The 2016 result was on one benchmark. A model can be great on the data it was trained and tested on and fall apart on a different microphone, an accent, background noise, or someone talking fast.

OpenAI’s Whisper paper in 2022 is basically about this problem. They trained on 680,000 hours of audio paired with transcripts collected from the internet, multilingual and messy, instead of a small clean dataset.⁵ The interesting comparison in the paper is against a model trained on LibriSpeech, a clean audiobook dataset. Both models score about the same on LibriSpeech itself. On a pile of other datasets, the Whisper model made 55.2% fewer errors on average.⁵ The authors also point out that models which “match humans” on clean audio still made roughly twice as many errors as a human on other datasets.⁵

That matches what using voice software used to feel like. It worked when you spoke clearly and slowly in a quiet room, which is to say it worked when you were not really talking like yourself. Robustness is what lets you talk normally.

The software on the other end stopped needing clean input

This is the part I think people underrate. Even a perfect transcript of how people talk is messy. We say “um”. We restart sentences. We say “no wait, I mean the other one”. A study of conversation by Bortfeld and colleagues found that these disfluencies go up when the thing you are talking about is harder to plan.⁶ That makes sense. When you are thinking out loud, you are literally planning as you go, and the transcript shows it.

Older voice software needed you to speak in commands. “Set a timer for ten minutes.” “Call Mom.” The words had to fit a template. If you rambled, it broke.

A language model on the other end changes that. You can say something like “okay so the thing I want to remember is, actually two things, one is the API key thing for the gateway, the other is to ask about the deadline” and it can pull out the two items. The software can work with some of that messiness, though extracting the wrong intent is still a real risk.

That is exactly how DFM Voice works. You tap the widget, the app is already listening, and you talk. An agent decides what to keep and saves it to your notebook in your own words.⁴ We did not have to make the user speak in a format. That design only makes sense because the model can deal with how people actually talk.

Where voice still loses

I want to be honest about this part, because I think a lot of voice hype skips it.

Editing

Voice is great for producing text and bad for changing it. Try fixing one word in the middle of a paragraph by talking. You have to say which word, where, and what to replace it with. On a keyboard and a mouse that is a click and a few keystrokes. With voice it is a small negotiation.

This connects back to the Ruan numbers. Speech had fewer corrections during entry but slightly more errors left over at the end.¹ My guess, and it is only a guess, is that part of the reason is that fixing things by voice is annoying enough that people let small errors go.

Code is the extreme case. Code needs every character to be right. Brackets, indentation, variable names that are not real words. I would not want to dictate a Python function symbol by symbol. What I would do is describe the change I want and let an assistant write it, then review it on the screen. That is a different workflow. Voice for intent, keyboard and eyes for precision.

Precision in general

Some things are short, exact and easy to mishear. A phone number. A name spelled a specific way. An email address. Saying “a as in apple” is slower than typing one letter. Anything where one wrong character breaks it is still keyboard territory for me.

Other people

The biggest limit is not technical. You cannot talk to your laptop in a library or a classroom. You probably do not want to talk your private journal entry out loud on a bus. Typing is silent and private, and voice is neither.

In practice I use voice when I am alone and the keyboard everywhere else, and I do not see that changing soon.

This is also why I do not think “voice replaces the keyboard” is the right frame. It is more like the mouse. The mouse did not replace typing. It took over the jobs it was better at.

I like this picture because it shows that same split. Talking was always faster for the person with the idea. Someone else turned it into clean text. The interesting thing now is that the someone else is software, and it is available to anyone with a phone.

What is still missing

Capturing speech has become much easier. Conversation still exposes problems that transcription alone does not solve. When you go from “I say something and the computer writes it down” to “I talk with the computer”, a different set of problems shows up.

Turn-taking

Humans are absurdly good at taking turns. A study by Tanya Stivers and colleagues looked at conversations in 10 languages, from major world languages to small indigenous communities. The most common gap between one person finishing and the next starting was 0 milliseconds. The mean across the whole dataset was 208 ms.⁷ Languages differed a bit, from a mean of about 7 ms in Japanese to about 469 ms in Danish, but everyone was fast and everyone avoided long silences and overlap.⁷

Now the weird part. Stephen Levinson and Francisco Torreira point out that just planning a single word takes about 600 ms, and a short sentence takes longer.⁸ So if gaps are around 200 ms, people must start planning their reply while the other person is still talking. They argue that listeners predict where the other person is going and get ready early.⁸

Most voice software does the opposite. It waits until you stop, waits a bit longer to be sure you stopped, then starts thinking. That waiting is what makes talking to a computer feel like a walkie-talkie. Too short a wait and it cuts you off mid-thought. Too long and it feels slow.

In the DFM post I said the hard part of a voice product is knowing when the person has finished, and that VoiceOS handled that for us.⁴ Building with it made me appreciate how much of “does this feel natural” comes down to a few hundred milliseconds.

Interruption

Real conversation has overlap. You say “mm-hm” while someone talks. You cut in with “wait, no”. Most voice systems are half-duplex, which means one side talks at a time, like a walkie-talkie.

There is research going after this directly. Kyutai’s Moshi, from 2024, is a full-duplex model. It treats the user’s audio and its own audio as two parallel streams instead of taking turns, so it can handle overlap, interruptions and little interjections. They report a theoretical latency of 160 ms and around 200 ms in practice.⁹ That is right around the human gap from Stivers. I have not built anything with full-duplex models myself, so I do not know how well this holds up in a messy real app, but it is the direction I would bet on.

State

This is the one I think about most. When I talk to a friend, they remember what we said ten minutes ago, last week, and the general shape of what I care about. Most voice interactions start from zero every time.

Text interfaces get away with this because the history is right there on screen. You can scroll up. With voice there is nothing to scroll. If the system does not remember, you have to repeat yourself, and repeating yourself out loud is much more annoying than copying and pasting.

DFM Voice is a small version of this. Your entries go into a notebook, and letters to your future self come back months later.⁴ The memory is the product. I think a lot of voice apps will end up looking like that: a voice front end on top of something that keeps state well. Honestly, the voice part may turn out to be the easier half.

What I would build next

What I want from voice is a place to put an idea before I’ve worked out how to say it neatly. I might change direction halfway through or remember an important detail at the end. A useful assistant would help me shape that into something I can review without making me plan the whole sentence first.

I’d still want to type when the exact wording matters. The interesting future is being able to move between speaking and editing without losing the thought in between. If the software could keep track of corrections and ask about the genuinely unclear parts, talking to a computer might start to feel less like issuing commands and more like working through an idea.

Sources