Simon Willison found his blog contained 1,856 tags, a volume too large for an LLM to process reliably in a single classification request. He turned to a method proposed by Doug Turnbull that avoids asking the model to select from a fixed list entirely. Instead, the system instructs the AI to invent new tags without reference to the existing vocabulary. Vector embeddings then compare these imagined terms against the actual tag corpus to identify the closest concrete matches.
This approach removes the constraint of forcing a model to choose from potentially irrelevant options. It allows the system to generate highly specific labels that better reflect the nuance of the content being indexed. The method relies on the mathematical similarity of text rather than the model’s ability to recall a pre-defined menu.
* Model generates novel tags without vocabulary constraints
* Vector embeddings match outputs to the existing corpus
* System retrieves the closest concrete tags automatically
