Simon Willison applied DSPy to evaluate and improve the system prompts for Datasette Agent, a tool that executes read-only SQL queries to answer data questions. He triggered an asynchronous research task in Claude Code using Claude Fable 5 to test GPT 4.1 mini and nano models against the existing prompt logic. The analysis identified that the schema listing provided only table names, which led to column-name guessing errors such as `page_count` and `o.order_id`. This specific advice caused the model to enter error-retry loops in baseline traces.
The practical outcome involves either including column names directly in the prompt’s schema listing or softening the instruction against calling `describe_table` when information is already available. This change addresses a concrete failure mode where the system wastes tokens on redundant schema introspection.
* Column names were missing from the initial schema listing
* The model guessed invalid identifiers like `o.order_id`
* Error-retry loops occurred due to redundant `describe_table` calls
