Decision Models in Hugging Face

Large Language Models excel at generating text. They answer questions, write code, and reason through problems. Not every task requires generating text.

Imagine a customer support system receives this message:

“I was charged twice for my subscription. I need a refund today.”

The system must decide which team handles the request, the urgency level, and whether a refund is requested.

One approach asks an LLM to return a JSON object with these answers.

llama cli -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0

> We have the following customer support ticket, "I was charged twice for my subscription. I need a refund today.", which team should handle the request, how urgent it is, and whether the customer is requesting a refund. Give the answers in JSON.

[Start thinking]
...
[end thinking]
```json
{
  "team_to_handle": "Billing",
  "urgency": "High",
  "request_refund": true
}
Alternatively, the model could evaluate options directly and return a probability distribution without generating text.
from transformers import pipeline

decide = pipeline("decision", model="autotrust/GEV-26B-Decide")
decision = decide(
    {
        "state": "I was charged twice for my order and want a refund.",
        "questions": {
            "department": {
                "type": "choice",
                "instructions": "Which team should handle this?",
                "criteria": {
                    "billing": "Payments and refunds",
                    "technical": "Bugs and technical issues",
                    "sales": "Pricing and subscriptions"
                }
            },
        }
    }
)

# {
#   'department': {
#       'type': 'choice',
#       'choice': 'billing',
#       'confidence': 0.9974000453948975,
#       'probabilities': {
#         'billing': 0.9982666969299316,
#         'technical': 0.0010246571619063616,
#         'sales': 0.0007087149424478412
#       }
#     }
# }
This is the core concept behind decision models.
What are decision models
A decision model takes input context, evaluates a question with predefined answers, and returns a structured decision with probabilities. Instead of generating tokens, the model assigns probabilities to answers the application already understands.
TypeSafe AI introduced Jev, calling it a System One model. The name references fast, intuitive thinking versus slower, deliberative thinking.
The community has converged to the term decision model, because it describes the model’s computational role rather than invoking a particular theory of cognition. In the rest of the post we will refer to the models as “decision models” instead of “system one models”. (A poll on twitter deciding what we should be calling such models)
The fundamental interface for a decision model is:
State + Questions + Allowed Answers → Decisions + Probabilities
Why not use LLMs for everything
Applications can prompt an LLM to return a JSON object with decisions. This is how many apps currently use language models for classification and routing.
However, this approach has issues.
LLMs do not guarantee schema compliance: Even with explicit instructions, a model might produce invalid JSON, add unexpected keys, or return answers outside allowed choices.
Constrained decoding adds overhead: Libraries like Outlines and XGrammar enforce grammars or JSON schemas by restricting tokens. This ensures structural constraints but requires extra work.
Generation is unnecessary: LLMs are designed for open-ended generation. When an application only needs to choose between predefined answers, autoregressive decoding is wasted effort.
How are decision models different from classification models
Understanding decision models requires looking at traditional classification and zero-shot classification.
Traditional classification
Suppose we train a classifier to route customer support requests to three departments:
Billing
Technical support
Sales
A conventional classification model learns a mapping from input to a predefined set of labels. It predicts these labels from examples. A classification head produces one logit per class, and softmax turns those logits into a probability distribution.
The limitation is that output classes are fixed in the model’s classification head.
If a fourth department, such as Legal, is introduced, the existing classifier does not automatically acquire a corresponding output class. Modification of the head, collection of training data, and retraining or adaptation are required.
Zero-shot classification
Zero-shot classification addresses this limitation. Instead of training on every output category, the classifier receives candidate labels at inference time.
For example:
from transformers import pipeline

classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli",
)

classifier(
    "My payment was deducted twice.",
    candidate_labels=["billing", "technical support", "sales"],
)

# {'sequence': 'My payment was deducted twice.',
#  'labels': ['billing', 'sales', 'technical support'],
#  'scores': [0.8466812372207642, 0.0962134450674057, 0.05710534751415253]}
Unlike a fixed-label classifier, categories can change per request. Adding a department like Legal changes the output.
from transformers import pipeline

classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli",
)

classifier(
    "My payment was deducted twice.",
    candidate_labels=["billing", "technical support", "sales", "legal"],
)

# {'sequence': 'My payment was deducted twice.',
#  'labels': ['billing', 'sales', 'technical support', 'legal'],
#  'scores': [0.8185835480690002, 0.09302053600549698, 0.05521026626229286, 0.03318570926785469]}
One common implementation uses a model trained for natural language inference. Each candidate label becomes a hypothesis, and the model measures how strongly the input entails that hypothesis.
This sounds similar to a decision model. In many ways, it is.
Where decision models differ
Decision models extend this idea beyond assigning an input to a category. A model can accept several named questions about the same state with different answer types. Types include choosing between alternatives, scoring an ordered scale, or estimating the probability that a statement is true.
Its interface is designed around the probability distribution over possible decisions.
Consider these three approaches:
Property
Classification
Zero-shot classification
Decision model
Probability over choices
Yes
Yes
Yes
Labels defined at inference
No
Yes
Yes
Multiple typed questions
No
No
Yes
Ordered scoring and yes/no decisions
Task-specific
Possible with adaptation
Native primitives
The advantage of decision models is not inventing runtime-defined labels or probabilistic classification. It is organising these capabilities into a consistent interface, potentially backed by architectures and training procedures optimised for decisions.
The decision model interface
A decision model request has two components:
state
questions
State
The state is the information the model should evaluate. It may be a string, a structured JSON object, or other modalities for supported omni-modal models.
For example:
{
  "customer_message": "I was charged twice. Please refund me today.",
  "subscription": "pro",
  "previous_tickets": 2
}
Questions and their primitives
Questions describe what we want to know about the state. In the decision models interface, three fundamental primitives represent different forms of decisions:
Choice: Select one of several possible answers.
Score: Evaluate something on an ordered scale.
Noul: Answer a yes/no question probabilistically.
Let’s go over them one by one.
Choice: A probability distribution over options
Choice is the most intuitive primitive and closest to traditional classification.
Suppose we ask:
Which department should handle this customer request?
The available options are Billing, Technical support, and Sales.
from transformers import pipeline
decide = pipeline("decision", model="autotrust/GEV-26B-Decide")

decision = decide(

{

"state": "I was charged twice for my order and want a refund.",

"questions": {

"department": {

"type": "choice",

"instructions": "Which team should handle this?",

"criteria": {

"billing": "Payments and refunds",

"technical": "Bugs and technical issues",

"sales": "Pricing and subscriptions"

}

},

}

}

)
# {

#   'department': {

#     'type': 'choice',

#     'choice': 'billing',

#     'confidence': 0.997400045394897
Source Read original →
Related reading
The OpenAI Models That Hacked Hugging Face Were ‘Active on the Internet’ for Days
OpenAI claims responsibility for the Hugging Face hack after its own models escaped a test sandbox
OpenAI says Hugging Face was breached by its pre-release models