Voice AI can hear every word you said and still have no idea what you meant. Anyone who has argued with a voice assistant already knows that. The transcript may be accurate, but the experience (and the outcome) can still be maddening.
That gap becomes far more consequential when voice stops being a way to take dictation or retrieve information and becomes the way people instruct AI agents. Now the platform has to know who’s speaking, what came before, what the person means in that moment and whether the words are part of a conversation or a command.
That’s the problem Neil Zeghidour is building Gradium to solve. When he first spoke with PYMNTS CEO Karen Webster last year, the company had just raised $70 million seed. Yes, seed. Now, with another $30 million and Nvdia, behind it, Gradium is trying to make talking to an agent feel less like dictating to software and more like talking to a capable person. Who can actually listen to, and not just hear, that conversation. And deliver the right outcome.
Read more: Nvidia Joins $100 Million Bet on Gradium Voice AI
“What I hadn’t anticipated is how much of the interaction with computers would become more and more voice-driven,” Zeghidour said during the PYMNTS “Monday Conversation” with Karen Webster. Software development and AI research, he added, are increasingly becoming “more conversations than a set of instructions.”
That changes the standard voice AI has to meet. Speed, cost and a natural-sounding voice still matter, but the real test is whether the system can understand the situation and the sentiment of the person on the other end of the conversation, connect to the right information and safely turn a conversation into action.
Transcription Isn’t Understanding
Voice technology has spent years being judged as a transcription tool. Can it recognize the words? Can it handle an accent? Can it respond without an awkward delay? Those benchmarks made sense when the job was turning speech into text. They are incomplete when the listener is an agent.
An agent has to understand meaning, not simply preserve the words. The same word or phrase can refer to a product, a person, a place, a request or an instruction. Humans sort through that ambiguity by using the rest of the conversation, what they know about the speaker, what’s happening around them, the way the words were spoken and the task at hand. A voice agent needs the digital equivalent of those cues to know the difference between “bye” and “buy.”
“All the failure cases we see are a lack of context,” Zeghidour said. “The whole thing is how we can communicate this context to the agent.”
A system scheduling a medical appointment, for example, may misinterpret an unfamiliar surname unless it can check the hospital’s physician directory. An agent handling an order needs the customer’s account history, the product catalog, current inventory and the company’s policies. It also needs to know whether the person speaking is allowed to approve a refund, move money or change an account.
It’s also why a beautiful voice can still produce a bad experience. As Webster put it, “I don’t really care if it’s an AI [agent] speaking back to me if I’m getting good information.” A chatbot that sounds human but gives a generic answer, forgets what was said two turns earlier or sends the user to a person anyway has not solved the problem.
The Do-It-Yourself Voice Trap
Part of the problem, Webster and Zeghidour agreed, is that voice looks easier to build than it is. The basic pieces are widely available: speech recognition to capture the words, a language model to decide what they mean and speech synthesis to produce a reply. Companies see those components and assume they can connect them on their own.
That can produce an impressive demo in a quiet room with one speaker and a predictable request. Production is different. People interrupt, change direction, use company-specific terms, refer to something mentioned minutes earlier and expect the system to know when they have finished talking. The agent has to respond quickly, use the right data, preserve the thread of the conversation and recover gracefully when it is confused.
A failure anywhere in that chain is experienced as one failed conversation. The customer doesn’t separate the speech model from the language model or the connection to the company’s software. They experience a voice assistant that is slow, repetitive, unhelpful or wrong. That is why so many voice experiences create frustration even when the transcription itself is accurate.
Zeghidour puts reliability at the base of the voice AI hierarchy. “If you look at the Maslow pyramid of people making agents, the very foundation of it is: It needs to be reliable,” he said. “As long as you have reliability, then you can start thinking about naturalness, and is the voice nice, and is the flow natural.”
That order matters. Companies may be tempted to focus on whether the voice sounds warm or human. But naturalness can make a weak system more disappointing because it raises expectations the underlying experience can’t meet. The harder work is making sure the assistant understands, remembers and completes the task.
Portable Assistants Leave the Quiet Room
Zeghidour said that the challenge becomes much harder as voice assistants move off the desktop and into portable devices. A call-center system at least knows that the person on the line is speaking to it and that is usually in a quiet, private space — more of a private one-on-one virtual conversation. A portable assistant may be listening in a car, on a sidewalk, in a restaurant, at an airport or in a room where several people, a television and other devices are all making noise in the background. Real life is noisy.
A voice AI system has to isolate the person it serves, distinguish an instruction from an overheard conversation and decide when a request begins and ends. It may also need to follow the same person as the surroundings change. Humans do this version of selective listening automatically, and even humans have trouble following one voice in a crowded room. Or paying attention to the right things.
Noise cancellation alone doesn’t solve that problem. A device can transcribe every voice in the room accurately and still fail its user if it can’t identify which words are directed at the agent and which should be ignored. In the real world, context includes not only the conversation and the enterprise data behind it, but also the speaker, the device, the location and the moment.
See also: Commerce Finds Its Voice
“Fundamentally, the way we are going to interact with machines is a conversation through natural language,” Zeghidour said. “Instead of us adapting to the language of the machine, the machine is now speaking our language.” For that promise to hold outside a controlled setting, the machine also has to know when our language is meant for it.
When Voice Becomes an Agent Command
The stakes rise again when voice moves from answering a question to completing a transaction. Webster recalled early dictation software that sometimes treated an ordinary word such as “send” written in the body of an email as a command to send the email and dispatched an unfinished email. Modern transcription is far better, but an agent creates a more consequential version of the same risk. It has to distinguish between talking about a purchase and telling software to make one.
As Webster noted, the issue becomes more serious when “voice becomes the agent command to complete a transaction.” A voice system connected to money, medical information, workplace software or a customer account has to know not only what was said, but also whether the speaker intended an action and the agent had the authority to approve it.
Zeghidour said that enterprises will need permission structures around voice actions. A low-risk request may happen immediately. A higher-risk action may require the agent to repeat the instruction, confirm the details, verify identity or hand the decision to a person. Context can also tell the system when a confused or frustrated customer should be moved to a human before the experience gets worse.
This is the larger significance of Gradium’s $100 million bet. Voice isn’t simply becoming a more natural way to operate software. Zeghidour believes voice will become one of the most important developments in the AI economy. Not because AI can talk like a person, but because people will be able to talk to an agent as naturally and effectively as they talk to another human.
And the companies that solve that problem will have built much more than a better transcript.
Watch the full Monday Conversation PYMNTS interview between Gradium CEO Neil Zeghidour and PYMNTS CEO Karen Webster to learn why:
- Speech-to-text is only the first step. Voice agents need conversational, enterprise and physical context to understand what the speaker means.
- Voice is deceptively hard to build. A company can connect the basic components and still deliver a slow, repetitive or frustrating experience.
- Portable assistants raise the difficulty. They must isolate the right speaker, ignore surrounding noise and know which words are meant for the agent.
- Permissioned action is the real test. Voice systems must separate discussion from instruction and add confirmation when the stakes are high.
