Google researchers have introduced StreamVibe, a video agent designed to decide for itself when to speak up. Tests show how difficult that remains for current AI models.
A team from Google, Stanford, Oxford, and several US universities is building an assistant that watches you cook or assemble furniture and speaks only when needed. With StreamVibe, they have created a video agent designed to understand vague requests such as “Please help me if I run into any difficulties.”
Existing video assistants are built for clear-cut commands like “Let me know when the water starts boiling,” so they rarely pick up on unspoken intentions. They also struggle to track what has already been done, which causes them to repeat advice or respond too late. When users change their requests midway, the assistants often stick to the original instructions.
Three building blocks for a more attentive assistant
To measure the problem, the team compiled a benchmark of about 4,100 examples from more than 183 hours of everyday life and software recordings. Gemini 2.5 Flash suggested when to offer help and what to say, while four human evaluators had to unanimously approve each example. The test covers three stages including clear instructions, vague goals, and requests that change over time. In one example, a user taking apart a toy excavator asks for progress updates and then switches to requesting a quality check after three minutes.
From there, the researchers trained StreamVibe-VL, a model apparently based on Alibaba’s Qwen3.5-9B that decides every second whether to stay quiet or speak. A second training stage uses reinforcement learning to make the model more reserved so it does not constantly interrupt. A separate memory system tracks which requests are pending and which steps are complete, though the model alone decides whether to speak and what to say.
All tested models lose track of the conversation in continuous operation
According to the researchers, StreamVibe-VL improves response timing and quality by 10.7 percent over other models, while the memory system is designed to boost proactive interaction by another 12.3 percent and cut down on unnecessary interruptions.
StreamVibe-VL generally beats its base model and other open-source models by a wide margin, though Google’s proprietary Gemini 3.8 Flash scores higher in several tests. According to the paper, the specialised training does not hurt general video comprehension.
Models often make the right call when they only have to decide at specific moments whether a prompt is appropriate. But when they process an entire video and have to build on their own earlier responses, performance across all tested systems plummets as errors compound, and a key content metric for the responses drops by up to 61.5 percent based on the system’s own history. In videos longer than five minutes, the models rarely pick the right moment and find it especially hard to track a changed instruction over a long stretch.
In a GIMP tutorial on converting a photo to black-and-white documented in the paper and on the project page, the comparison model JoyAI says the finished image has already been saved even though the user has not opened it yet, while StreamVibe-VL gives the next step at the right moment, such as “Colors” and then “Desaturate.” In a step-by-step guide for making rolled toast with jam, other models just describe what is happening on screen, such as the bread being flattened, while StreamVibe-VL advises cutting off the crust first.
The researchers acknowledge that real-world situations are likely more complex than their test setup, and future work will examine how user preferences and the environment influence each other over time.
Google has been working on AI systems that actively analyse videos for a while now. In September, Google introduced agent-based video analysis for Gemini, where the model itself decides which scenes to examine more closely. With Gemini 4 Argon, Google also recently showed how AI can track longer action sequences in robot videos and spot errors. StreamVibe’s assistant is designed to understand what is happening at any moment and recognise when its help is actually needed.
On the ProactiveBench benchmark, AI models often guess or refuse to answer rather than ask specific questions when visual information is missing. A separate video comprehension benchmark revealed major weaknesses in spatial and temporal reasoning. And for autonomous AI agents, the conversation itself can hurt performance, as Microsoft researchers have shown.
What it means
For people using software or watching tutorials, the current generation of assistants often interrupts too often or misses the point entirely. StreamVibe aims to stop that noise. It tries to wait until a user is actually stuck before offering help. The research shows that even with better training, models still struggle to keep track of a conversation once it goes on for several minutes.
