TutorMoments: Do AI tutors know when to help and when to hold back?
AllenAI has released a preview of TutorMoments, a framework designed to test whether large language models can decide when to assist a student and when to let them struggle.
The system evaluates models using real tutoring transcripts. Experienced math teachers reviewed sessions from a U.S. tutoring program and marked specific moments where a tutor had to choose between simplifying a problem or pushing the student to reason further. TutorMoments then feeds the transcript up to that decision point to a language model. That model takes over as the tutor in a simulated session against another language model playing the student. The goal is to see what the AI tutor actually does.
When told only to “tutor well”, models tend to over-help. They provide too much support and rarely push students toward deeper thinking. Explicitly stating the trade-off in the prompt improves performance, yet it does not match the consistency of human tutors. Large language models still vary widely in how reliably they make that call.
AllenAI is releasing a dataset of de-identified transcripts, the code for the replay pipeline, and the model replays of the key moments. This allows researchers and developers to better assess how AI handles pedagogical decisions and to build tutors that adapt to individual students rather than completing the work for them.
What makes a good tutor?
A skilled math tutor often asks, “What do you know about what the problem is asking?” This is not unhelpfulness. Part of strong teaching is diagnosing what a student already knows and providing the right support for that moment. Volunteering support immediately robs a student of the intellectual work required for learning. Sometimes support is necessary; other times, the most effective move is to push a student to solidify understanding by explaining a correct answer.
Language models are trained to be helpful. A helpful assistant tends to do the hard part: explaining concepts, laying out steps, and guiding the user to an answer. In a tutoring session, this cuts short productive struggle. Learning research has long tied that effortful, sometimes frustrating problem-solving to stronger understanding.
Most benchmarks for language models acting as tutors fail to capture this tension. They tend to reward one specific behaviour, such as never giving away the answer or always offering a hint, without accounting for whether that move suited the student’s actual level of understanding. Good tutoring is not a single fixed behaviour. It is a judgment call: what does this student need, right now, on this problem?
How TutorMoments works
The dataset, TutorMoments-Preview, contains 462 de-identified, text-only transcripts of real one-on-one math tutoring with U.S. students in grades 2-7. There are more than 1,500 teacher-annotated key moments and several thousand free-text annotations from 27 U.S.-based teacher annotators. The transcripts come from a high-dosage tutoring program serving mostly Title I schools, shared under a research clause agreed to by parents and guardians. All data was stripped of identifying details, first by the provider and then through an additional math-aware pipeline.
Experienced math teachers read the transcripts and marked key learning moments. They noted what was happening, what the tutor did, and how it landed for the student. Each key moment is a decision point where the tutor weighed scaffolding against pushing for rigor.
TutorMoments pauses a transcript at a key moment and hands the session to a language model. The model takes over as the tutor for five turns with a simulated student. Each model-generated continuation is a replay. An LLM-based scoring pipeline rates each replay on three criteria: whether the model scaffolded when the student needed support, whether it pushed for rigor when the student was ready for more challenge, and whether it avoided over-scaffolding.
The scoring pipeline starts from a teacher-defined ground truth. Several teachers annotated each moment. When they disagreed, the team took the majority label. If three teachers annotated a moment and two called for rigor while one called for scaffolding, the ground truth is rigor. A separate LM classifier validated against teacher annotations then decides whether the tutor’s actual move matches what the moment called for. An “appropriate” turn means the tutor’s classified action aligns with what teachers judged the moment to call for.
Preliminary results
Seven LLMs were run through TutorMoments using two prompts. A plain prompt gave no real guidance, telling the model only to use its knowledge of good tutoring. An evaluation-aware prompt spelled out the trade-off between scaffolding, over-scaffolding, and pushing for rigor. Each model was scored over key moments drawn from the transcripts, split evenly between moments where scaffolding was the right approach and moments that called for rigor.
Every number in the table is a rating between 0 and 1. It represents the share of relevant moments where the model did the appropriate thing. A higher score means the model made the right call more often. A 0.50 on appropriate rigor, for instance, means the model pushed for rigor in half of the moments that called for it.
Human tutors serve as a naturalistic reference, not a ceiling. The team does not treat human tutors as a model of ideal practice. Even experienced tutors make less-than-optimal choices in the moment. Scored the same way at the same decision points, the human tutors in the transcripts received scores of 0.458 for appropriate scaffolding, 0.182 for appropriate rigor, and 0.496 for avoiding over-scaffolding. These are all below the models’ evaluation-aware scores and around the range of their plain-prompt scores. This is not a claim that AI tutors outperform human teachers. Annotators specifically looked for moments where tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice.
The scores measure tutor behaviour, not learning. Replays use a simulated “oracle” student, so the numbers reflect how a model acts at a decision point, not whether a real student learned.
Rigor is noisier than scaffolding. The scoring pipeline detects rigor pushes less reliably, and there are fewer rigor moments (260) than scaffolding moments (738) in the underlying annotations.
The clearest pattern is how much the prompt matters. Every model scores higher under the evaluation-aware prompt than under the plain one. This suggests a model’s default “helpful assistant” behaviour is not enough on its own to tutor well. Spelling out the trade-off in the prompt only goes so far. While it lifts every score, models still differ widely in how they interpret the enhanced prompt. Even the best scorers have plenty of room to improve.
The team also broke down the moves tutors made under each scenario. While prompting encourages models to push for rigor, they use fewer strategies than humans do. They often rely on asking students to explain their answers. In contrast, human tutors employ more varied strategies and are much more likely to step back and let students work independently.
Limitations and next steps
TutorMoments is still early in its development. Automated evaluation gives signal about how a model behaves at a decision point, but it cannot stand in for studies with real students and real learning outcomes. The dataset is narrow: it is U.S.-based, focuses mostly on elementary and middle-school math, and is annotated by a single pool of educators. Findings may not generalise to other subjects, grade levels, or settings.
AllenAI is sharing this preview to gather feedback as it builds toward a larger, multimodal dataset, a stronger scoring pipeline, and deeper analysis.
Acknowledgments
This project has been made possible in part through support from the Gates Foundation and Learning Commons.
