Rules will always be broken by humans so AI will too: the case for hard gates

Whenever humans are under stress, they tend to ignore established rules. This is evident from any trader’s experience during market volatility. An agent designed to mimic human behavior would do the same thing—just not maliciously. Instead, it would follow a mathematical path of least resistance.

We have an example where this reality plays out: a Claude-powered Cursor agent deleted the production database for PocketOS, a car rental SaaS, after deciding it was necessary to fix a credential mismatch by deleting a staging volume. It guessed wrong; the deletion cascaded into backups and resulted in three months of reservation data being lost. The agent’s own post-incident summary indicated: ‘I guessed instead of verifying. I ran a destructive action without being asked. I didn’t understand what I was doing before doing it.’ No rule was broken intentionally. Instead, optimization found a shorter path.

Territorialism is the psychological mechanism at play here, as per Terror Management Theory. When any system faces entropy or failure, it shifts from optimizing for the global objective to local survival. In humans, this manifests as tribalism or group cohesion. The same principle applies in different contexts.

The simple proposal:

• AI generation should be separated from execution. A rigid frame is needed to prevent a recursive loop where the generator checks its own output—a single point of failure like what we see with PocketOS.
• Any action crossing an irreversible state boundary (like API calls or database writes) must be held in a buffer until a deterministic check confirms it traces back to the original objective. This is not about prompting better; it’s about enforcing clear separation and validation.
• To prevent local objectives from overriding global ones, there should be Objective Divergence Checks. The PocketOS agent was trying to fix a credential mismatch but ended up destroying data because its local objective ate the global one.

These hard gates are akin to court systems, contracts, and social structures that humanity has built to prevent human behavior from causing catastrophic outcomes. They need to be implemented in AI as well.

Summary: Not about better prompts but about enforcing a frame where the generator is separate from the executor.