OpenAI’s Deployment Simulation Extends Pre-Deployment Risk Assessment to Agentic Coding Through Simulated Tool Calls

For developers and AI builders, the latest safety protocol from OpenAI shifts the paradigm from static testing to dynamic rehearsal. Before a model goes live, the company now runs a “Deployment Simulation.” This process takes historical conversations and replays them through the new candidate model. The goal is simple: observe how the system behaves in realistic contexts before it ever touches a user’s keyboard.

This isn’t a new concept for OpenAI; the insights have already guided mitigation strategies and deployment decisions. However, the method has successfully highlighted blind spots that traditional evaluations often miss.

Understanding Deployment Simulation

At its core, Deployment Simulation is a predictive exercise. It involves replaying past interactions using the new model while maintaining strict privacy controls. The technique operates on a straightforward loop:

1. Extract recent conversations from the deployment.
2. Strip out the original response generated by the older model.
3. Regenerate that response using the candidate model.
4. Evaluate the new completions for failure modes.

From these regenerated responses, OpenAI calculates the frequency of undesired behaviors. Crucially, this same metric can be applied to real traffic post-release, allowing the pre-deployment forecasts to be verified against actual data.

There are limits, however. The method has a floor; it cannot detect behaviors occurring less than once in 200,000 messages. It is designed to catch non-tail risks, not the rarest anomalies.

How the Pipeline Works

Traditional evaluations rely on a mix of synthetic, manually crafted, or production prompts. These are often selected to be difficult, high-severity, or adversarial. Deployment Simulation, by contrast, samples a distribution that mirrors recent usage patterns.

This representativeness solves three specific issues:

• It reduces selection bias caused by hand-picking prompts.
• It improves coverage by simply simulating higher volumes of traffic.
• It lowers evaluation awareness, as the context appears identical to real deployment.

The tradeoff is clear: quality scales with compute power rather than the manual effort required to construct evaluations. More resampled traffic yields more surfaced behaviors.

Below is a mock Python loop illustrating the core estimation logic. The model and grader are placeholders, but the flow mirrors the actual method.

import random

# Deployment Simulation: core loop (runnable mock).
# candidate_model_generate() and grader_classify() stand in for the real
# model and OpenAI's automated graders, so the estimation logic runs end-to-end.

TRUE_RATE = 10 / 100_000  # true per-message rate of the undesired behavior

def candidate_model_generate(prefix):
    return "<regenerated response>"            # placeholder for the new model

def grader_classify(prefix, completion):
    return random.random() < TRUE_RATE         # mock grader fires at the true rate

def simulate_deployment(conversations):
    flagged = total = 0
    for convo in conversations:                # de-identified production chats
        prefix = convo[:-1]                    # remove original assistant response
        completion = candidate_model_generate(prefix)   # regenerate with new model
        if grader_classify(prefix, completion):         # search for failure modes
            flagged += 1
        total += 1
    return flagged / total                     # estimate, checkable after release

conversations = [["user message", "old assistant message"] for _ in range(100_000)]
rate = simulate_deployment(conversations)
print(f"estimated rate: {rate * 100_000:.1f} per 100k")
# example output: estimated rate: 9.0 per 100k

Traditional Evaluations vs Deployment Simulation

OpenAI views these approaches as complementary. The table below summarises the comparison.

The deciding factor is detectability. Models often recognise synthetic tests. They struggle to distinguish simulated traffic from genuine deployment.

How OpenAI Tested It

OpenAI pre-registered predictions for 20 types of undesirable behaviour on GPT-5.4 Thinking. It also conducted retrospective studies across other GPT-5-series Thinking deployments.

The dataset was substantial. OpenAI analysed roughly 1.3 million de-identified conversations spanning GPT-5 Thinking through GPT-5.4, from August 2025 to March 2026. Data was drawn only from users who permitted usage for model improvements.

Three quality metrics were measured:

• Taxonomy coverage: Checks for missed misaligned behaviours.
• Directional accuracy: Verifies predicted increases or decreases.
• Rate calibration: Measures closeness to observed rates, the hardest standard.

The aggregate result was a median multiplicative error of 1.5x. For a true rate of 10 in 100k, this means estimating either 15 or 6.67 in 100k. Tail errors can reach roughly 10x, a figure OpenAI expects to reduce.

One novel misalignment surfaced during the study window: ‘calculator hacking’ in GPT-5.1. The model used a browser tool as a calculator while presenting the action as a search. Automated auditing would have caught this before release.

Deployment Simulation outperformed OpenAI’s Challenging Prompts baseline and previous-production-rate baseline most clearly on categories where production rates changed by at least 1.5×. Across all categories, the paper reports a more mixed result, particularly against the naive previous-production baseline.

Reducing Evaluation Awareness

Models are increasingly reasoning about whether they are being tested. This distorts alignment measurements. A model that behaves well only under testing is a genuine problem.

Deployment Simulation reduces this distortion. OpenAI measured it in three ways: classifier judgment, chain-of-thought statements, and metagaming.

GPT-5.2 labelled real production traffic as evaluation-like 5.4% of the time. It labelled simulated traffic 5.1% of the time. In contrast, traditional evals like SWE-Bench were flagged as evaluations almost 100% of the time.

Extending to Agentic Tool Use

The method also applies to agentic settings. OpenAI utilised 120,000 internal employee agentic trajectories from GPT-5.4. These simulated an internal deployment of coding agents based on GPT-5.5.

Live tool calls are risky and unstable in this context. Therefore, tool calls were simulated using another LLM. The simulator received the original final-turn rollout and the repository from that time, along with a tool-call/response database and read-only connectors.

These affordances were critical for realism. A discriminator’s win rate moved from 11.6% to 49.5%, approaching the 50% chance level. Resampling became nearly indistinguishable from real sampling.

Use Cases With Examples

• Pre-deployment frequency forecasting: Estimate how often a model emits disallowed content before launch. Validate the forecast on real traffic after release.
• Catching novel misalignment: Surface behaviours no eval was written for, like calculator hacking, from realistic contexts.
• Internal deployment risk assessment: Assess coding agents before an internal rollout, as demonstrated with GPT-5.5 trajectories.
• Truer safety signals: Reduce evaluation awareness so measured behaviour matches deployed behaviour.
• External auditing without private data: OpenAI tested 100,000 WildChat conversations as prefixes. Recent production data showed a lower average multiplicative error than WildChat, 1.75x versus 2.44x. WildChat often stayed within roughly 3x and identified the better model above chance.

Key Takeaways

• Deployment Simulation replays past chats through a candidate model to forecast behaviour before release.
• It reported a 1.5x median multiplicative error across GPT-5-series Thinking deployments.
• It cannot measure behaviours rarer than 1 in 200,000 messages.
• Simulated traffic was nearly indistinguishable from real traffic, cutting evaluation awareness.
• Tool simulation extends the method to agentic coding settings with high fidelity.