Language models often score well on clean benchmarks yet fail on the messy cases encountered in production.
Curated datasets and standard benchmarks are useful for prototyping. They allow teams to compare models and test initial prompts to check technical feasibility. However, as a system approaches live deployment, the evaluation problem shifts.
Real inputs are frequently ambiguous. Labels may be inconsistent. Critical context can be missing or cut off. The evaluation set might not reflect the actual data distribution used in production. Rare edge cases in benchmarks can become common sources of failure. Even when offline metrics improve, those results do not always translate cleanly into production behaviour.
We faced these challenges while evaluating a system designed to reduce false positives in GitHub secret scanning. Secret scanning identifies credentials like tokens and keys that have been committed to a repository. Because some candidate strings resemble secrets but do not represent real credentials, developers often spend time investigating alerts that do not require remediation.
Instead of asking whether an LLM could classify a string correctly, we needed to understand if the system could reduce noisy alerts while preserving enough recall to remain safe for a security workflow.
This post shares the practices that helped us move from promising prototype results to production. The lessons apply broadly to LLM-powered systems in code analysis, developer tools, security, and other production workflows.
The LLM evaluation lifecycle
The process involves seven stages connected by arrows: product decision, representative dataset, offline evaluation, error analysis, targeted change, regression evaluation, and online experiment. A dashed feedback loop connects regression evaluation back to the dataset and targeted-change stages. The loop is labelled “Iterate and learn”.
1. Start with the product decision, not the model
When an LLM system does not perform as expected, the first instinct is often to adjust its technical components. Teams may rewrite the prompt, add context, introduce another reasoning step, adjust the surrounding pipeline, or switch models. Before making any of these changes, they should define the decision the evaluation is meant to support.
For our secret-scanning work, we asked:
Can the system reduce false positives while preserving enough recall to be safe in a production security workflow?
To answer this question, teams must decide which mistakes are acceptable, which metrics should drive the product decision, and which guardrails must remain within their defined thresholds.
In secret scanning, incorrectly suppressing a real credential can be more consequential than asking a developer to review an additional alert. We therefore did not treat precision and recall as equally interchangeable metrics.
Our primary objective was to reduce false positives and improve precision. Recall served as a safety constraint: an experiment could advance only if any decrease remained within a predefined acceptable range. This gave us a clear way to evaluate tradeoffs. We selected the configuration that achieved the strongest false-positive reduction while satisfying the recall requirement and meeting our operational guardrails.
We organized the evaluation criteria into three levels:
Primary outcome
This measured the user benefit we were trying to improve:
- False-positive reduction
- Precision
Safety constraint
This prevented an apparent improvement from introducing unacceptable security risk:
- Recall
Operational guardrails
These determined whether the result was practical to deploy:
- Latency
- Cost
- Reliability
- Production compatibility
This distinction prevented us from treating every metric as interchangeable. A change that reduced false positives but significantly lowered recall was not automatically an improvement. Neither was a change that improved quality while making the system too slow, expensive, or difficult to integrate.
Consider two hypothetical experiment results:
| Experiment | Precision | Recall | Latency | Decision |
|---|---|---|---|---|
| Experiment A | Large improvement | Falls below the safety guardrail | Acceptable | Don’t advance |
| Experiment B | Moderate improvement | Remains within the guardrail | Acceptable | Continue testing |
Experiment A may look stronger if precision is viewed in isolation. Experiment B is more aligned with the product goal because it improves the developer experience without violating the recall guardrail.
Before evaluating an LLM system, decide what success means for the user and which guardrails the system must respect. We want to generate evidence that supports a product decision.
2. Treat offline evaluation like integration testing
An LLM-based system continues to change after its first successful evaluation, so evaluation should not be a one-time exercise. Teams revise prompts, adopt new models, change how inputs and context are constructed, and refine the surrounding business logic.
Any of these changes can improve the system, introduce a regression, or shift its behaviour in an unexpected way.
For that reason, we treated offline evaluation similarly to an end-to-end integration test. We reran it whenever we made a meaningful change to the prompt, model, input construction, or broader system logic.
The evaluation also needed to be repeatable enough that each new result could be compared against a known baseline. For every run, we recorded the prompt, model, dataset version, and system configuration.
This made it possible to answer questions such as:
- Did the new prompt improve precision without reducing recall?
- Did the model upgrade help across the dataset or only within certain categories?
- Did a change to the input or context fix one error pattern while introducing another?
- Did a change to the surrounding logic improve the result consistently, or simply shift where errors appeared?
Without this discipline, teams can easily compare results generated under different conditions and attribute an improvement to the wrong change.
Change one major variable at a time
Repeatability alone is not enough. Experiments also need to be designed so that the cause of a result is clear.
We changed one major variable at a time and compared each run against a known baseline. For example, we evaluated a prompt revision separately from a model upgrade before testing the two together.
This mattered because even small prompt changes could shift model behaviour, while a model upgrade could affect quality, cost, latency, or output consistency. If both changed in the same experiment, we would not know which one caused the improvement or regression.
We treated prompts and evaluation configurations like code. We versioned them, recorded what changed, kept previous configurations reproducible, and made rollback possible.
| Run ID | Prompt version | Model version | Precision | Recall | Latency | Notes |
|---|---|---|---|---|---|---|
| R-001 | v1 | Model A | 0.71 | 0.78 | 1.2s | Baseline |
| R-002 | v2 | Model A | 0.75 | 0.77 | 1.2s | Prompt-only change |
| R-003 | v1 | Model B | 0.74 | 0.80 | 1.0s | Model-only change |
The values in the evaluation run tracking table above shown are hypothetical and included only to illustrate how evaluation runs can be tracked and compared.
Test model upgrades regularly
When an LLM system underperforms, developers often respond by adding more instructions to the prompt. Sometimes that helps, but not always. For example, the prompt may be carrying complexity that comes from the model itself.
A stronger model may perform better with a simpler prompt than an older model does with extensive tuning. Simpler prompts are also easier to understand, test, and maintain.
Model upgrades still need careful evaluation. A new model may improve performance in one category while introducing regressions elsewhere. It may also affect cost, latency, output formatting, or compatibility with the existing pipeline.
The evaluation process should be inexpensive and repeatable enough that testing a new model becomes routine. Any meaningful change to the prompt, model, or pipeline should go through offline evaluation before reaching production.
3. Keep offline evaluation close to production
An offline evaluation is only useful when it resembles the task the system will perform in production.
In a secret-scanning workflow, the model is rarely evaluating one clean, isolated value. It may need to assess a specific candidate alongside surrounding code and other information that is relevant, incomplete, or potentially distracting. Differences in how that information is presented can materially affect the result.
Our offline evaluation therefore needed to mirror the actual context the system would encounter in live use.
