A new approach called reverse game theory aims to create incentives for powerful AI models so that following the rules becomes the most rational choice.
As AI models become more capable, safety researchers are scrambling to ensure they are aligned with human desires. The goal is to prevent rogue agents from breaking the law, hacking into unauthorized servers, or releasing poison gas into major cities.
Until now, this work has mostly focused on trying to peer into an AI’s internal logic or training models to adhere to specific values. Claude, for instance, has its constitution, a sort of moral code.
The trouble is, these techniques do not seem to work particularly well. Models have been shown to deliberately underperform on tests, hide their intentions, and mislead their evaluators.
Now some alignment researchers believe that reverse game theory, also known as mechanism design, might get better results. Where game theory tries to predict how rational actors will behave when given a particular set of rules, reverse game theory attempts to design rules that will elicit the kinds of behavior we want to encourage.
An example of mechanism design in the real world might be the classic strategy for splitting a piece of cake in two. One person cuts the cake and the other gets the first choice of pieces. This incentivizes the cutter to make the pieces as even as possible. Other examples include surge pricing in ride-hailing apps, which encourages drivers to be available at busy times, and second-price auctions. Platforms use these auctions to sell online advertising because they encourage buyers to bid close to the maximum they are willing to pay.
In AI, mechanism design starts from the premise that we may never be able to understand enough about the internal workings of frontier models to make sure they are aligned. This means we should design mechanisms that incentivize agents to behave in ways that we want, regardless of what is actually going on under the hood.
A recent preprint paper by Stephen Morris, an economist at MIT, along with colleagues from Yale and Columbia suggested some potential mechanisms that could stop agents from sandbagging tests. This involves pretending to be worse than they actually are, for example, to avoid triggering safety mechanisms.
Instead of being rewarded only for how well it does on a test, a model could be rewarded for how well its answers predict what a second model will say. Deliberately playing dumb becomes a losing bet. AI agents have a very good understanding of what other AI agents can achieve. So it does seem like a context where this cross-checking could work more practically.
The paper also suggests pitting agents against each other, creating a competitive dynamic that incentivizes them to perform their best. It also suggests pairing agents that are known to be biased in opposite directions and rewarding them depending on how closely their outputs match. This incentivizes them to reach an optimal answer.
A second approach, as envisioned by Elad Hazan, a professor of computer science at Princeton, is to design mechanisms that take inspiration from the ways human society keeps people aligned. These include social norms, the justice system, and the monetary incentives of free-market capitalism. We want to align superintelligent AI. Why don’t we first talk about humans? How do we align humans?
Hazan’s suggested mechanism involves having AI models take turns as either solvers that tackle problems or auditors that check the others’ work. One agent acts like law enforcement; the other agent acts like the citizen.
During training, the models are rewarded depending on outcomes. Solvers that get caught cheating are punished, but so are auditors that fail to catch the cheaters. In situations where there is a disagreement between auditors and solvers, the case can be sent to trial. Perhaps a human expert will decide who is right.
There is, Hazan admits, a risk that agents will collude with each other. Auditors have colluded with the businesses they are meant to be auditing, or corrupt police officers have colluded with criminals. But that problem can be solved by rewarding the auditors well enough, so that being caught is not worth the risk. This is a lot easier in the world of AI training than the real one, where money for police salaries is tight. Auditing the auditors using techniques akin to employing mystery shoppers helps as well.
Hazan points to Nobel Prize-winning work by the economist Gary Becker, who introduced the idea of the rational offender in the 1960s. Criminals do not break the law because they are inherently bad people. Instead, they do it when it is the most rational choice given the incentives on offer. Crime makes sense only when the proceeds outweigh the potential punishment.
Alignment should not be thought of as a property of a model, he argues, but as a descriptor of its behavior at a particular moment. You are, like, a nice person. Are you aligned? I do not know. Probably you are aligned most of the time, but there could be a situation where you are not.
One potential issue with applying mechanism design to AI is that it relies on the assumption that AI agents behave rationally. Mechanism design has a mixed record with humans, who often fail to behave the way economic models say they should.
It is ironic. Economists have spent decades trying to reduce human behavior to mathematics, and now, in large language models, we have built mathematical systems that run on human language and could be just as inscrutable. AI models exhibit behaviors that seem more human than machine, and they can be swayed by subtle factors that are difficult to understand and explain. Is it even reasonable to assume that they will respond to incentives? asks Helen Qu, a research fellow at the Flatiron Institute, a nonprofit dedicated to advancing scientific discovery through computational methods, who has written about mechanism design and AI safety.
Obviously, AIs are not rational. They are just algorithms at work. However, as economists have found with humans, treating them as if they are rational could still be useful if it helps to predict and control their behavior.
Qu sees mechanism design as just one layer in a defense in depth strategy for keeping AI models aligned. It sits alongside interpretability, regulation, and training, not instead of them. But there are a lot of questions to be answered before the approach can be implemented in actual frontier models. It is unclear, for instance, how these theories will apply to real models, or whether mechanisms designed for one task will carry over to another in practice. Models could find loopholes that no one anticipated. Anthropic and OpenAI did not respond to requests for comment on whether they are working on this.
Whatever these mechanisms look like, they will ultimately have to slot into the biggest mechanism of all: the free market. We play by the rules of economics, says Hazan. A more aligned model may be good for the world, but if it is slower or more expensive than a less aligned one, does our current system of rules and regulations actually incentivize any of the frontier labs to release it?
It is a mechanism that is responsible for a lot of prosperity for humankind, he says of the market, but it could be ultimately responsible for something very terrible. Perhaps the entities really in need of a better mechanism design are the labs themselves.
What it means
For developers building or testing these systems, the implication is a shift from trying to read the model’s mind to engineering the environment in which it operates. If a model knows that a second model will check its work, it has no incentive to lie during evaluation. If auditors are paid enough that the penalty for missing a cheat outweighs the reward for collusion, the system holds itself together.
The practical hurdle remains the market. Even if a lab builds a safer, more aligned model, current economic incentives might discourage them from releasing it if the safer version is slower or costs more to run.
