Cisco AI Introduces FAPO: Pipeline-Aware Prompt Optimization With Step-Level Failure Attribution and Claude Code Orchestration

Getting prompts right is still the hardest part of shipping reliable LLM applications. Small wording changes can swing accuracy by 20 percent. What works on a few examples often breaks at scale. When a multi-step pipeline returns a wrong answer, finding the failing step means inspecting intermediate outputs by hand.

FAPO automates the search for better prompts

Cisco AI introduced FAPO to address that bottleneck. FAPO stands for Fully Automated Prompt Optimization. It is a Claude Code-driven system that optimizes LLM pipelines from baseline prompts to target accuracy. You supply a dataset and an initial prompt. FAPO then evaluates, classifies failures, proposes variants, validates them, and iterates. The whole loop is orchestrated by Claude Code agents. The project ships open source under Apache 2.0, and also supports Codex as the optimization agent.

How the system works

FAPO is a multi-tenant evaluation and optimization framework. A tenant is a self-contained optimization project. Each tenant directory holds one task’s prompts, dataset, chain definition, scorer, and config. Tenants stay isolated, so unrelated tasks optimize side by side without interference.

The core engine is named hephaestus and is domain-agnostic. It handles evaluation, chain execution, and scoring. Chains are LangGraph state graphs that process each test case. Out of the box, FAPO supports three providers: OpenAI, Baseten, and SageMaker.

The one input you must bring is a dataset. It is paired inputs and expected outputs that define success. FAPO splits it into a validation set and a held-out test set. The validation set drives iteration; the test set is used only for a final one-shot evaluation. From a task description, Claude can scaffold the rest: the initial prompt, the chain, and the scorer.

The optimization loop

Once the pieces exist, FAPO runs a closed loop until target accuracy is reached. Each cycle runs six stages:

1. Evaluate — run the chain on the dataset, collect per-case scores and step-level outputs.
2. Attribute — classify failures by root cause using rule-based heuristics plus LLM analysis.
3. Propose — generate a variant targeting the dominant failure cluster.
4. Review — an independent agent validates the proposal for scope compliance and data leakage.
5. Compare — accept the variant only if it improves on the previous best, otherwise reject.
6. Iterate — continue until target accuracy is reached or the optimization budget is exhausted.

The system works at three escalating levels. Prompt edits are lowest cost and tried first. Parameter changes adjust config values like retrieval_k or temperature. Structural changes alter chain topology, such as adding a self-reflection node or switching to a ReAct pattern. FAPO exhausts one level before escalating to the next.

Step attribution sorts failures into four classes. Retrieval failures return empty or irrelevant content. Cascading failures begin when an early step produces empty output. Format failures hide the correct answer inside text the scorer cannot parse. Reasoning failures occur when good inputs still produce a wrong conclusion. Format and reasoning issues are prompt-addressable. Retrieval and cascade issues are structural-addressable.

Guardrails keep the optimizer from overfitting. It inspects only training-split cases, while validation and test expose aggregate scores only. Every variant is a new immutable file, never edited in place. An independent reviewer checks each proposal before it runs.

Performance against GEPA

Cisco team evaluated FAPO against GEPA (Generalized Evolutionary Prompt Architecture), a state-of-the-art prompt optimization method. GEPA uses evolutionary search with genetic operators to optimize prompts for multi-step pipelines. Both systems started from identical baseline pipelines and prompts. FAPO could escalate to structural changes when attribution found bottlenecks. GEPA was limited to prompt-level optimization.

The comparison spanned six benchmarks and three task models: GPT-4.1-mini, GPT-5.4-mini, and Gemma 3-12B. Claude Opus 4.6 served as both FAPO’s orchestrator and GEPA’s reflector. Scores below are averaged across the three task models.

FAPO won 15 of 18 model-benchmark comparisons, with a mean gain of +14.1pp over GEPA. On HoVer and IFBench, where FAPO escalated to pipeline changes, it won all six model-benchmark pairs. The mean gain there was +33.8pp. On the four benchmarks without structural changes, FAPO still won 9 of 12 through prompt optimization alone. AIME was the only benchmark where GEPA led, by 3.1pp. The gap is smaller than the standard deviation across stochastic trials.

A capability comparison shows the design difference reported by Cisco. Every row below reflects the source description of the two systems.

Use cases

FAPO targets multi-step LLM pipelines, not single prompts. A few concrete examples:

• Multi-hop question answering: A chain retrieves documents, extracts facts, reasons over evidence, and formats an answer. In Cisco’s documented walkthrough, a multi-hop QA chain rose from 39.3% to 70.3% validation exact match across two iterations. Attribution then flagged the remaining failures as retrieval-limited, signaling a structural fix. Separately, on the HotpotQA benchmark, FAPO reached 68.3% test accuracy versus GEPA’s 61.8%.
• Instruction following: On IFBench, format-constraint failures pushed FAPO to escalate beyond prompts, reaching 80.7% test accuracy.
• Classification: A software-name-to-category task can be scaffolded by Claude Code, then optimized to exact-match targets.
• ReAct agents: An MCP workflow extension optimizes a tool-calling ReAct agent using trajectory scoring and LLM-as-Judge scoring.

Getting started

The fastest path is to let Claude Code create the tenant files. From the repo, describe your task in plain English, then add a JSONL dataset. Each line is one test case with case_id, task_type, context, expected, and metadata:

{"case_id": "1", "task_type": "qa", "context": {"question": "What is the capital of France?"}, "expected": {"answer": "Paris"}, "metadata": {}}
{"case_id": "2", "task_type": "qa", "context": {"question": "What is 2 + 2?"}, "expected": {"answer": "4"}, "metadata": {}}

A scorer compares the chain output to the expected answer. It implements validate_case to catch bad data early and score_case to return a composite score:

from hephaestus.scoring.scorer import Scorer as BaseScorer

class Scorer(BaseScorer):
    def validate_case(self, case, scoring_profi