Generative AI control systems introduce unpredictability to robotics, creating a new safety hurdle for manufacturers and users
Handing control over to a generative AI model is the current trend in robotics, but this architecture lacks the predictability of traditional algorithms. Companies must now find a way to guarantee that a new humanoid robot will not cause harm.
Dr Ding Zhao, who directs the Safe AI lab at Carnegie Mellon University, has spent his career addressing this issue. He has now joined forces with Kyle Wong, a veteran startup executive, and Simo Rachidi, a machine learning engineer, to launch Safeworld. The company aims to solve the safety problem inherent in these new systems.
“The safety challenge is a combination of advanced generative AI probabilistic evals — how do you underwrite the risk of a probabilistic system?” Zhao says. “The second part that is really hard is the trust part, and you need both to deploy a robot.”
Safeworld is emerging from stealth today with a seed round of more than $12 million. The funding is led by Shine Capital and a16z Speedrun, with additional investment from Box Group, the Carnegie Mellon University Endowment, Innovation Endeavors and SV Angel.
“The time to build an industry safety standard is now while robots are being designed and deployed,” Jonathan Lai, a partner at a16z Speedrun, told TechCrunch. “By the time you have robots in households colliding with kids and causing safety incidents, that’s way too late.”
The company specialises in evaluating robotic control systems in simulations populated with realistic human models. This mirrors the challenge faced by companies like Tesla or Wayve, which must ensure vehicles respond appropriately to surprising incidents on the road. Zhao argues this will be more difficult for robots because they work in unstructured environments and each facility has different safety standards.
“One of the most common areas is if there is a blind corner in this particular factory,” Wong said. “What is the speed or what is the stopping distance that you need to make sure that this robot will not collide with a particular human? If a human is carrying boxes, for example, will the robot detect the human or not?”
To answer that, Safeworld will build a digital version of the corner in a model like Genesis or MuJoCo. They insert a simulation of the robot it is evaluating, driven by its real software, and then run thousands of scenarios where human models encounter the robot. That is harder than it seems, per Zhao, because people are unpredictable.
“Tripping and falling is also a good example of something that we do a lot of testing with the simulation,” Wong said. “Otherwise, you would have to go and trip and fall for the robot, which is like a hard thing to be doing all the time.”
There are definite similarities between the platform Safeworld is building and the tools used internally by robot builders. The founders believe, however, that beyond their specific expertise, robot-makers will want a third party to validate their work. This allows competitors to share information about safety cases.
“A lot of people are underestimating one how hard some of these edge cases are going to be to solve,” Zhao said. “It is not the robot in the vacuum, in the demo, that we are worried about. It is the robot that is deployed at scale, with people who potentially never operated a robot before.”
Vishal Dugar, the CTO of Gritt Robotics, is developing the AI brain for robots that currently help workers install photovoltaic panels at industrial-scale solar farms. His company aspires to take on more complex construction tasks. Gritt Robotics is partnering with Safeworld as they develop their safety simulations.
“The difficulty with most of our systems is it’s very hard to formally prove it by doing some math, writing some equations, and saying yeah, the system is verified to be safe,” Dugar says. “It necessarily has to be done empirically.”
His robots operate alongside human workers, and ensuring that the robotic arm does not hit them is obviously top of mind. To verify that in practice requires considering all kinds of potential scenarios.
“Humans have many kinds of appearances,” Dugar points out. “Their bodies can be in different configurations. They could be kneeling, standing. They could be tripping and falling potentially. They could be crouching. They could be running. You have to respond to all these behaviors that humans could potentially exhibit on these sites, along with the variety of variations in human appearance, you know, clothes, size, shape, height, skin color, everything else.”
It is still early days for both Safeworld and generative AI in robotics. The company is still figuring out the best model for its product — a platform for external users, or a services based approach? — but the team is confident they are taking on the right problem.
“We’ll probably be the first profitable company in this field,” Zhao says. “Because if anyone wants to deploy, they need to pay us to handle the situation.”
What it means
Robot builders face a choice between internal validation and external verification. Safeworld proposes a third-party solution where manufacturers pay to have their systems stress-tested against unpredictable human behaviour. This shifts the burden of proving safety from mathematical proof to empirical simulation, potentially creating a new cost barrier for deploying AI-driven robots in real-world settings.
