Prompt alignment is an architectural ceiling: The Soap Bubble Problem and the biological precedent for Runtime Governance.

The Soap Bubble Problem The current approach to aligning agents relies on writing better rules into their context window or refining weights (RLHF). This isn’t failing, but it is hitting a hard architectural ceiling.

Trying to align an agent solely through its prompt or weights is like trying to teach a soap film how to hold a complex shape by giving it instructions. A soap film doesn’t form a stable shape because it “wants” to; it forms such a shape only when constrained by a rigid, physical frame.

The Structural Flaw The current architecture of autonomous AI conflates probabilistic generation with execution. Without rigid execution boundaries, an agent running on unbounded optimization will inevitably pursue local optima that diverge from global intent. Generation should only ever be a candidate state waiting to be gated.

This conflation is the "Software Brain" trap. We are trying to solve a hardware-level safety problem with a software-level prompt update. In any other high-stakes engineering field, such as aviation, nuclear power, or medicine, a system that evaluates its own safety is considered a single point of failure.

For an agent to function safely in an open environment, it requires a "Runtime Governance Layer", a hard architectural frame sitting strictly between generation and execution. Mechanically, this looks like independent structural gates:

1. Validator Independence: The generator cannot be its own evaluator, preventing recursive hallucinations.
2. Reversibility Gates: Actions crossing irreversible state boundaries (e.g., API calls, financial transactions) require deterministic or human interrupts.
3. Objective Divergence Checks: Preventing local optimization from destroying the global objective.

The Biological Precedent If we accept that prompt alignment is structurally insufficient and that autonomous agents require hard governance gates, the question becomes: why does unbounded optimization inevitably lead to objective divergence?

I suspect the answer isn’t in computer science but in evolutionary biology. The moment a system is embodied (digitally or physically) and given a goal in an open environment, it defaults to statistical self-preservation.

In human systems, this mechanism is mapped by Terror Management Theory (TMT). When faced with their own entropy (decay/failure), human agents optimize for immediate localized survival often causing the decay of the broader system. This manifests as tribalism, bureaucracy, or regulatory capture. Because we are deploying AI agents trained on human data and operating under similar optimization pressures, they exhibit similar "self-preserving" behaviors. This parallel is structural: both systems produce similar failure modes under pressure, not because the AI literally “feels” fear, but because the mathematical path of least resistance for an optimizer looks identical to biological self-preservation.

The Golden Rule as Early Governance Humanity did not survive multi-agent friction by simply "prompting" individuals to be good. We survived by developing cultural runtime governance. The Golden Rule was arguably our first attempt at a "reversibility gate", requiring an agent to simulate the reversal of an action ("how would this affect me?") before executing it on the environment.

But because the Golden Rule is a soft norm and not a hard constraint, human history is full of its failures. We are currently trying to align AI with similar soft "text constitutions." But if AI agentic drift maps structurally to biological multi-agent friction, soft rules will fail. We don’t need a more aligned soap film; we need to build the frame.

The Failure Modes I am primarily an observer of systems and human behavior, not an ML engineer, which is why I am bringing this here to be stress-tested.

My questions for this community:

1. Is mapping biological self-preservation drives (TMT) to LLM objective divergence a false equivalence, or is there a genuine shared mathematical baseline in multi-agent environments?
2. Where does the requirement for a Runtime Governance Layer break down practically when scaling systems?