Last week, a team at a Cambridge, Massachusetts startup showed robot arms stacking cups and sweeping blocks into bowls after watching a short video.
The arms learned these chores without specific training for each task. In one demonstration, a robot instructed to sweep a block into a bowl used a dustpan as a brush when the brush was removed from the scene.
In another case, a two-armed robot watched a clip of someone unzipping a purse to remove banknotes. It then unzipped a different purse and carefully took out the notes. When it could not grab the money with its right gripper, it switched to its left hand to get a better angle. One engineer nearby noted that the robot never did that before.
“This is exactly the kind of thing people were really excited about with GPT-3,” Pete Florence, Generalist’s cofounder and CEO, told me. He was referring to OpenAI‘s large language model released in 2020. “You could take that model and just prompt it to do a new task and it would have a real shot at doing it.”
Generalist focuses on teaching robots about the physics of the world, which mirrors the intuitive sense humans have from an early age. This approach may help the model transfer what it learned in one scenario to another.
Some demos reminded me of how children improvise when shown a task. Researchers were often surprised by what the robot decided to do. One chose to sweep up items with a banana when it was placed in front of it.
Physical intelligence is something still largely lacking in machines. The way babies learn so efficiently about their world may offer important insights for AI researchers.
I met Florence and Andrew Barry, cofounder and CTO, in a conference room overlooking teams of people doing robot training with special grippers on their hands. The company’s other cofounder and chief scientist is Andy Zeng. The trio previously worked at Google DeepMind and Boston Dynamics on advanced hardware and robotic models.
Traditionally, training an AI-powered robot to do different tasks has meant feeding thousands of examples into the model. This is a notoriously imperfect kind of learning. A robot will struggle with the task if you change something as simple as the lighting.
Generalist and some other robotics startups are investing heavily in a general robotic model trained by humans. The company builds special gloves resembling robot pincers that have cameras attached to them. People use these gloves to perform different chores. I saw a crate piled high with several hundred of these grippers destined for workers in Mexico and elsewhere.
Florence and the team are cagey about exactly what recipe they are using to train the robots. They say the company has already gathered a huge amount of high-quality training data. In contrast to some other companies chasing smarter robots, they have built their AI models entirely from scratch rather than relying on an open-source language model.
Danfei Xu, a roboticist at Georgia Tech who is familiar with Generalist’s work, says that the startup stands out among companies chasing more general robot models. “They have pushed this to the extreme, and they’ve done a really good job executing,” Xu says. Besides gathering a huge amount of high-quality data, he says, “they are excellent roboticists, and they have done really good science.”
Xu also says that the stuff Generalist has demo’d so far suggests that they have an eye on deploying robots in real commercial settings. “They are the closest to something that’s deployable,” he says.
“Generalist’s data approach is collecting physical interaction data at large scale without tying it too closely to one particular robot,” says Karen Liu, a roboticist at Stanford University who also knows the company. “Their strongest results suggest that this bet may be working.”
Generalist says the learning skills of its models are not yet all that reliable. A robot is only able to complete a task it has been shown about 59 percent of the time, on average. Ideally, its success rate would be somewhere upwards of 99 percent. It also seems unclear how well these skills will generalise to every imaginable task or setting.
Even so, the potential for robots to quickly learn skills in manufacturing seems huge. One of Generalist’s engineers seemed to discover this late one recent evening. A video that captured the episode shows the engineer stacking small cups on the table in front of a two-armed robot, just to see what the machine might do. The robot suddenly joined in, grabbing and stacking other cups with its two grippers. As the robot finished stacking the cups into one neat pile, the engineer began yelling to no one in particular, delighted by the maneuver.
What it means
Workers in warehouses and factories could eventually train machines by showing them a few examples rather than programming every possible scenario. This reduces the need for massive, static datasets and allows robots to adapt when the environment changes. However, the current success rate of about 59 percent means these systems cannot yet handle the full complexity of real-world commercial environments without significant improvement.
