Picture a video call with someone who stares at you, perfectly still, while you tell your best story. No nod, no smile, nothing. Then they burst out laughing three seconds before the punchline.
That's roughly where AI-generated "listeners" are today, and our new NeurIPS'26 paper called GLARE (Generating Listening Heads with Appropriate REactions)[1] sets out to change it.
Talking is solved-ish. Listening isn't.
AI has become very good at making faces talk, but a conversation has two people in it. While one talks, the other is constantly giving small signals: a nod that says "go on," a smile at the funny part, a frown that says "wait, what?" These reactions are how we tell each other we're paying attention.
What makes them hard is timing. A reaction at the wrong moment feels worse than no reaction at all. Laugh too early and you look like you weren't listening. Nod too late and you look distracted.
The researchers found three reasons AI listeners struggle:
- 📼 No good training data. Plenty of conversation videos exist, but nobody had labeled when the listener actually reacts.
- 🤖 Models learn to wiggle, not respond. Existing systems learn general head movement, so the listener looks alive without really engaging with what's being said.
- 📏 Nobody measures what matters. The usual metrics ask "does this look realistic?" rather than "did it react at the right moment?"
How GLARE tackles it
Step 1: Build a proper dataset. The Atmee AI team went through about 147 hours of real two-person conversations and marked more than 64,000 listener reactions in six flavors: nodding, head shaking, smiling, laughing, frowning, and looking surprised. Each one records exactly when it starts and ends. The result is a large, labeled record of how people react while listening.
Step 2: Teach the model to hear how things are said. What someone says matters, but how they say it matters a lot too. We tend to react when a speaker's voice rises with excitement, stresses a word, or shifts in emotion. GLARE uses an audio AI model to flag these "something just happened in their voice" moments and passes them along as hints, a bit like a friend nudging you: "this is the good part, react now."
Step 3: Grade it on reactions. GLARE is rewarded for making the right reaction at the right moment, not just for producing a nice-looking face. It's like judging an actor not just on looking natural, but on hitting their cues at the right moment.
Here's how the pieces fit together:
Prosody Conditioning: listening to the voice, not just the words
Prosody is the music of speech: pitch, loudness, emphasis. It's the difference between a flat "that's great" and an excited "that's great!", and it's often what prompts a listener to react. GLARE asks an audio AI model to flag the moments where the speaker's voice gets expressive, then passes that timeline to the generator as a hint about where reactions belong.
Temporal Reaction Loss: checking the reactions as it learns
As it learns, GLARE also has to say, moment by moment, which reaction it's making (nodding, smiling, laughing, frowning, head shaking, surprised, or none). Those answers are compared with the labeled dataset, and every missed or mistimed reaction counts against it. That way it can't get by on generic movement. It has to learn what each reaction is and when it should happen.
Together: prosody tells the model where to pay attention, and the reaction loss makes sure it responds correctly.
A new judge for listeners
The team also built a better way to score AI listeners. The metrics ask practical questions:
- R-F1: Did it make the right reaction?
- R-tIoU: Did the timing line up?
- R-ATD Did it jump the gun? Reacting early is penalized more than reacting a beat late, since that's what feels weird to real people.
- R-FID Does the reaction itself look convincing?
So... does it work?
Yes, pretty convincingly. The Atmee AI team compared GLARE with five earlier methods (best in bold):
RealTalk dataset
| Method | R-F1 ↑ | R-tIoU ↑ | R-ATD ↓ | R-FID ↓ |
|---|---|---|---|---|
| L2L | 0.45 | 0.51 | 78.3 | 24.7 |
| DIM | 0.58 | 0.55 | 133.1 | 23.0 |
| ViCo | 0.43 | 0.57 | 92.5 | 26.1 |
| ListenFormer | 0.33 | 0.65 | 73.4 | 23.1 |
| DyStream | 0.54 | 0.69 | 63.2 | 15.3 |
| GLARE | 0.59 | 0.70 | 57.4 | 15.2 |
Seamless dataset
| Method | R-F1 ↑ | R-tIoU ↑ | R-ATD ↓ | R-FID ↓ |
|---|---|---|---|---|
| L2L | 0.41 | 0.55 | 112.9 | 35.7 |
| DIM | 0.30 | 0.50 | 157.3 | 34.4 |
| ViCo | 0.39 | 0.52 | 140.3 | 37.9 |
| ListenFormer | 0.37 | 0.51 | 128.7 | 35.3 |
| DyStream | 0.41 | 0.55 | 110.8 | 30.8 |
| GLARE | 0.46 | 0.58 | 102.1 | 28.9 |
- 🎯 GLARE reacts at the right time more often. It had more correct reactions and tighter timing across the board.
- ✨ It also looks better. Improving the reactions didn't cost visual quality. GLARE came out ahead there too.
- 🗿 No more statues. In side-by-side comparisons, several older methods mostly produced frozen faces. GLARE's nods and smiles matched what the real listener did.
- 🧩 Both additions pull their weight. Rewarding correct reactions gave the biggest boost. The voice hints added extra polish on top.
Why you should care
As AI avatars show up in virtual assistants, customer service, games, and video calls, they'll spend a lot of their time listening. An avatar that nods in the right places and smiles at your joke will feel far more human than one that stares blankly.
GLARE isn't the final word. People react differently to the same moment, and word meaning and personality aren't modeled yet. But it's a clear step toward AI that seems to actually be listening to you.
References
Try it
Talk to an avatar that listens
Sign up in minutes. No download. Browser-based.
