In the race to build better large language models, volume is no longer the sole metric of success. The real challenge lies in ensuring that the training data contains dense, structured learning signals. While general web text, code repositories, and mathematical datasets provide a broad foundation, task-seeded synthetic Q&A generation adds a crucial layer of precision. These synthetic examples are compact and task-oriented, featuring a clear information need, constrained response spaces, and explanations that explicitly link evidence to conclusions. In a 100 billion token continuation experiment on the Nemotron-3 Nano model, this approach boosted MMLU-Pro scores by 1.8 points, average code performance by 1.9 points, and commonsense understanding by 1.6 points. Most notably, it drove a massive 11.1 point gain on the GPQA dataset, while average math scores remained stable.
This article outlines a specific workflow for generating task-seeded synthetic Q&A, designed for the Nemotron-family training runs, including the Ultra and Super iterations. The process treats public task training splits as “capability seeds” rather than data to be memorised. It generates new, task-aligned examples, enriches them with reasoning traces and relevant knowledge, and filters them into curated datasets. Crucially, held-out evaluation and test data are never used in the generation phase, ensuring no data leakage. Downstream training recipes then decide how to blend these synthetic datasets with the broader corpus.
At A Glance
| Element | Value |
|---|---|
| Seed source | Public task training splits available through lm-eval-harness |
| Scale | Approximately 70 tasks and 700 subtasks |
| Data types | Similar questions, answer-enriched samples, reasoning and context traces |
| Verification | Schema checks, format validation, deduplication, and majority-voted answer checks |
| Training use | Late-stage Nemotron-family training, including Ultra and Super workstreams |
| Main result | Gains on MMLU-Pro, code, commonsense, and GPQA in a 100B-token Nemotron-3 Nano continuation |
Generation Pipeline
The workflow operates as a compact loop: collect training-split seeds, normalise heterogeneous task records, generate new examples, enrich the answers, and filter the resulting data. Internally, the team used roughly 70 public task datasets from lm-eval-harness, covering about 700 subtasks. For each task, only suitable training splits were used as SDG seeds; held-out test data was excluded, as were tasks lacking appropriate training data.
The seed pool covered both knowledge-intensive and reasoning-intensive tasks:
| Seed group | Approximate coverage | Purpose |
|---|---|---|
| Knowledge-intensive tasks | 39 tasks, ~300 subtasks, ~3M seed samples | Improve factual, scientific, multilingual, and domain-specific QA behaviour |
| Reasoning-intensive tasks | 34 tasks, ~400 subtasks, ~1.5M seed samples | Improve analytical reasoning, logical reasoning, math, code, and commonsense reasoning |
For Nemotron Ultra and Super pretraining, a license-compatible subset of the generated data was selected, suitable for commercial model training.
The end-to-end process consists of five stages:
- Collect seed tasks. Enumerate available
lm-eval-harnesstasks, group them by output type, and retain only those with suitable training splits. - Normalise records. Since each
lm-eval-harnesstask defines its own fields and formatting in YAML, records are converted into a unified JSONL-style schema. Multiple-choice tasks yield the question and candidate options; generative tasks yield the prompt plus any provided context. - Generate similar examples. Given a seed example, the generator creates a new question that preserves the underlying capability while altering the content.
- Enrich answers. The generator solves the generated questions and appends the final answer alongside relevant reasoning, knowledge, or context.
- Filter and package. The pipeline applies schema checks, format validation, deduplication, and task-specific answer validation where possible. Multiple-choice data is easier to verify directly; generation-style data requires more cautious handling.
A key formatting choice involves storing semantic answer text rather than just option labels when feasible. For instance, writing the answer as dirt trapped under the fingernails provides a clearer training signal than simply writing B.
Why Task-Seeded Data?
Public task datasets are imperfect, but their training splits contain compact examples of how information is requested, constrained, and resolved. They capture useful correlations between task framing, domain knowledge, reasoning depth, candidate answers, and final response forms. A model may ingest abundant raw text during pretraining and still benefit from synthetic data that makes those correlations explicit.
Task-seeded synthetic data bridges this gap by converting public task training splits into data generation templates. Using only suitable training splits from broad task families, the system generates new examples that preserve the useful properties of the source interaction:
- Task framing: whether the example asks for selection, generation, classification, or explanation.
- Answer structure: multiple-choice options, short answers, free-form responses, or format-constrained outputs.
- Domain and context: science, commonsense, factual knowledge, math, code, multilingual QA, or reading comprehension.
- Difficulty and reasoning depth: whether the example requires a direct fact, a comparison among alternatives, or several reasoning steps.
- Explanatory signal: task-relevant knowledge, reasoning, or context that helps connect the question to the answer.
This approach exposes the model to reusable reasoning and knowledge-use patterns across task families, without tying the dataset to the surface format of a single source.
Why Use Broader Seed Tasks?
A useful way to interpret this pipeline is through transfer learning across task families. Many improvements do not come from learning a single task’s surface format. They come from strengthening reusable behaviours that appear across many tasks: identifying the information need, applying relevant domain knowledge, separating plausible alternatives, following response constraints, performing multi-step reasoning, and grounding a final answer in the right context.
Consequently, the system does not generate from a narrow set of task formats. Instead, it collects a broader set of training-split seed samples from lm-eval-harness to cover many neighbouring capability regions. A science QA seed can aid commonsense physical reasoning. A logical reasoning seed can assist with careful alternative comparison. A math or code seed can help with multi-step planning even when the final application differs. The goal is positive transfer learning across task families, while reducing the risk that the model simply learns the quirks of a single data source.
This motivation aligns with earlier evidence in Nemotron Nano pretraining. Using AGIEval training data improved MMLU-Pro, suggesting that structured Q&A data from one task family can improve behaviour outside the original source family. The broader seed collection used here extends that idea: rather than relying on one task source, it uses many training-split task families so that transferable reasoning, knowledge-use, and answer-selection behaviours have more opportunities to emerge.
Why Add Context And Reasoning?
The answer alone is often a weak training signal, especially for science, commonsense, and multi-step reasoning examples. Adding task-relevant knowledge or reasoning traces gives the model a path from question to answer and helps it learn why plausible distractors are incorrect.
The PIQA-style example in Figure 2 illustrates this distinction in a compact setting. The generated question can be answered with the correct option alone, but the answer-generation variants add the definition, historical context, and distractor analysis that make the record a stronger learning signal.
“The PIQA-style seed leads to fresh similar questions, and one generated question is expanded into two answer-enriched records.”
In an internal ablation comparing with-context versus no-context variants, the context-enriched version delivered stronger numbers on several knowledge- and reasoning-heavy evaluations:
| Evaluation | No context | With context | Change |
|---|---|---|---|
| ARC-Challenge | 91.89 | 92.24 | +0.35 |
| CommonsenseQA | 80.02 | 80.26 | +0.24 |
| PIQA | 82.86 | 84.44 | +1.58 |
| WinoGrande | 79.87 | 80.51 | +0.64 |
| AGIEval-en CoT | 63.16 | 69.32 | +6.16 |
| GPQA-Diamond CoT n-shot | 34.85 | 45.96 | +11.11 |
| MMLU-Pro 5-shot | 64.45 | 66.89 | +2.44 |
| MBPP+ sampled | 73.77 | 74.82 | +1.05 |
Training Use
The task-seeded synthetic data was mixed into late-stage Nemotron-family training. In one 100 billion token continuation experiment on the Nemotron-3 Nano model, adding newly synthesised task-seeded data improved several capability groups:
| Metric group | Before | After | Change |
|---|---|---|---|
| MMLU-Pro | 64.8 | 66.6 | +1.8 |
| Average code | 7 Source Read original → Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise. More in AI Music   |