For makers and artists relying on generative tools, the latest research suggests that simply training a model on domain-specific data is no longer enough to guarantee clean, usable output. While supervised fine-tuning (SFT) improves fluency, it often fails to stop the model from getting stuck in repetitive loops—a failure mode known as text degeneration. A new study demonstrates that a second training stage, Direct Preference Optimization (DPO), can systematically suppress this error without needing human preference labels.
Turning Model Failures Into Training Data
In April, the research team released DharmaOCR, a specialized optical character recognition model designed for structured document extraction. The accompanying paper benchmarks leading vision-language models, both open-source and commercial, on a specific task: extracting Brazilian Portuguese text. A key metric tracked was the text degeneration rate, defined as the frequency with which a model enters a repetition loop rather than producing a valid transcription.
Testing across various open-source families revealed that vanilla models had degeneration rates ranging from below 1% to over 33%. Supervised fine-tuning improved results for most models, yet rarely achieved production-ready levels. The data points to a structural ceiling: standard fine-tuning optimizes for correct outputs but does not explicitly penalize repetition loops.
A second training stage, applied after SFT on the same documents using the same models, eliminated text degeneration in every family tested without exception. The average reduction was 59.4%, with the best case reaching 87.6%.
Figure 1: DPO reduced degeneration relative to SFT in every family tested - average reduction of 59.4%, peak of 87.6% (Nanonets-OCR2–3B: 1.61% to 0.20%). The direction is invariant; only the magnitude varies.
This second stage was Direct Preference Optimization (DPO). While most published DPO applications focus on chat alignment—training models on human judgments of helpfulness or safety—OCR tasks are objective. There is no conversational context. However, a clear preference signal exists: a correct transcription is chosen, and a degeneration loop is rejected. DharmaOCR used this binary signal to construct a DPO training set, testing the technique not for alignment, but as a direct mitigation tool for a specific failure mode.
The training signal originated from the model itself, specifically from the outputs it produced when it failed. Converting a failure mode into a training signal addresses the structural nature of the error rather than the model’s parameters.
The Loop Survives Fine-Tuning
Why SFT hits a ceiling on degeneration remains an open question, but the leading theory points to loss granularity. SFT trains token by token, evaluating each prediction in isolation. Consequently, a repetition loop is never penalized as a completion-level failure. DPO inverts this logic. The training signal is the full output—chosen or rejected—which allows a degenerated completion to be explicitly labelled as the wrong outcome, not just a sequence of locally probable tokens.
When a training objective maximizes the likelihood of observed sequences, it concentrates probability mass in the regions of distribution space those sequences occupy. A model entering one of these high-probability attractor regions during inference assigns elevated probability to the same token at the next step. This increases the probability further, sustaining the loop until the sequence hits the maximum token limit. Text degeneration is the output of this geometry: a self-reinforcing repetition loop that an autoregressive model cannot exit without external intervention. It is not purely a decoding artifact. The attractor involves the training objective, the learned distribution, and how probability mass concentrates during inference—a systems-level failure rather than a failure localized to any single component.
The geometry of this failure is visible at the token level.
Figure 2: When a token dominates its own conditional distribution, every sampling step deepens the attractor. The decoder samples from this geometry; it does not determine it.
Inference-layer interventions—such as repetition penalties, temperature adjustments, or early-abort logic—operate on the sampling step. They contain the symptom without touching the distribution that produces it. The attractor persists.
Supervised fine-tuning moves the distribution closer to the task domain. For a structured generation pipeline, this means training on domain-specific documents, in the target language, with the required output format. The model gains fluency with longer sequences, constrained syntax, and domain vocabulary. What SFT does not do is attack degeneration directly. Its objective—maximizing the likelihood of observed sequences—has no term that penalizes repetition loops. The failure mode is simply outside the scope of what the training signal optimizes for.
One model family in the DharmaOCR benchmark showed an unexpected pattern: a vanilla degeneration rate of 0.60% rising to 3.23% after SFT, before a subsequent DPO stage brought it to 1.41%. It is a single data point—an exception, not a rule—and overstating the evidence to treat it as proof of a mechanism would be unwise. What it does illustrate is that SFT does not reliably reduce degeneration. Capability and degeneration resistance can move independently.
The distinction matters structurally. SFT and DPO are not interchangeable training stages performing the same operation at different intensities. SFT closes the distance between the model’s prior distribution and the task domain. What it does not do is target degeneration as an objective—its effect on the failure mode is incidental, and the benchmark results show it is not consistent. The attractor that produces degeneration is not a problem with the model’s proximity to the task—it is a problem with the shape of the distribution space the model now occupies.
Addressing that geometry requires a training signal built specifically to point the model away from its own failure modes. For a structured, non-conversational task with no human preference labels and no conventional “helpful versus harmful” distinction, constructing that signal is a design decision.
The Design Decision: Degenerate Outputs As Rejection Pairs
The DharmaOCR pipeline’s contribution to DPO methodology is specific: it used the SFT model’s own degenerate outputs as the rejected examples—not as noise to remove, but as the negative training signal the optimization needed.
DPO requires preference pairs: a chosen output and a rejected output for the same input, with a quality difference clear enough for the optimization to learn from. In chat alignment, human annotators produce those judgments—rating responses as more or less helpful, accurate, or safe. Structured generation tasks have no equivalent annotation source. An OCR pipeline either produces a correct transcription or it does not. Quality differences exist, but they are not produced by human preference rankings—they are produced by the task’s own criteria for correctness.
The DharmaOCR pipeline identified a preference signal that structured generation tasks already produce: the range of outputs the SFT model generates in inference. A model capable of performing a structured task is also capable of failing at it in characteristic ways. Those failures—outputs that enter the degeneration attractor—are not noise to filter. They are the most informative negative signal available.
The paper implemented this on 23,726 training documents, generating multiple candidate responses per document with the SFT model and scoring each with an automated LLM judge. The pipeline is shown below.
Figure 3: The critical design decision is not in the pipeline's structure - it is in what the pipeline preserved: outputs displaying text degeneration were deliberately labeled as rejected examples, not filtered out as low-quality noise.
The conventional response when degenerate outputs appear in training data is to remove them. They are low-quality signal; filtering produces a cleaner dataset. The DharmaOCR approach inverted this logic. Degenerate outputs were deliberately retained as the rejected examples in each (chosen, rejected) pair, because they represent exactly the failure mode the DPO stage was designed to suppress. Removing them would have discarded the clearest target available.
The paper describes this as “preference-guided implicit unlikelihood”—the model is trained not only toward better outputs but away from a specific class of failure. Where SFT maximizes the likelihood of high-quality outputs, the DPO stage simultaneously penalizes outputs displaying the degeneration attractor geometry. The direction of the optimization is explicit in a way SFT alone cannot achieve.
Degenerate outputs are particularly well-suited as rejection examples because they represent a consistent failure mode rather than varied low-quality outputs. A transcription that misses words is low quality, but its failure is case-specific. Repetition loops, by contrast, appeared persistently across documents and model families even after SFT—a pattern consistent with a failure mode that likelihood-based optimization does not reliably correct. DPO applies its loss differently: at the completion level, with explicit rejection signals. The post-hoc analysis cannot establish causality, but the evidence suggests that what SFT’s objective leaves unresolved, DPO’s may address.
This approach requires no specialized annotation infrastructure—only a model capable of producing both acceptable and identifiable-failure outputs, and a scoring model to label preference pairs. A rule-based mechanism could detect repetition loops mechanically—but it could not identify which outputs represented high-quality transcriptions worth preserving as chosen examples.
The scoring model does both: it flags degeneration as the rejected output and validates clean extractions as the chosen one, keeping the model’s extraction capability intact while the DPO signal penalizes the failure mode. Whether the resulting training signal successfully moves the distribution in the intended direction—and whether it does so consistently across architectures—is the evidence question.
Consistent Across Five Model Families
The DPO stage reduced text degeneration in every model family tested—with reductions ranging from 37% to 88% and an average of 59.4% relative to SFT alone. The result held across architectures, parameter scales, and starting degeneration profiles that differed by more than one order of magnitude. One case in the dataset saw degeneration increase after the SFT stage before DPO corrected it. That case does not complicate the consistency. It confirms the mechanism more directly than any of the others.
Figure 1 shows the three-stage degeneration rate for each of the five model families tested: Vanilla, SFT, and SFT+DPO. In four of the five families, degeneration falls at each stage. The fifth family’s bars move differently—and that difference is the most analytically important data point in the study.
The Qwen2.5-VL-3B result, read carefully, is not a complication. It is a confirmation. The model’s vanilla degeneration rate was 0.60%—not because it was stable, but because it was too generic to produce long structured outputs at all. The model was not entering the degeneration attractor because it was not attempting the task seriously enough to find it.
SFT changed that. After domain adaptation, Qwen2.5-VL-3B became capable of the task—producing longer, more structured outputs with the domain vocabulary and format the pipeline required. That capability brought it into proximity with the degeneration attractor for the first time. Its degeneration rate rose to 3.23%.
This is the mechanism made visible: the act of improving capability can inadvertently create the conditions for failure, and only a targeted optimization stage can correct it.
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.
