Logan Scott

Research › Project MAZE

Automated Generation and Scoring of MAZE Reading Comprehension Assessments

University Scholars Program research project, 2026–2027 · Also supported as a UF AI² Center AI Scholars project.

Project title
Automated Generation and Scoring of MAZE Reading Comprehension Assessments (“Project MAZE”)
Principal investigator
Dr. Walter L. Leite, Ph.D., Professor (faculty mentor)
Institution
University of Florida
Department
School of Human Development and Organizational Studies in Education (SHDOSE), College of Education / Virtual Learning Lab
Time on project
September 2025 – present (one year), continuing through the 2026–2027 University Scholars Program year
My role
Research Assistant and Lead Developer; lead author on the scoring and reliability work
Dissemination
Paper accepted through peer review to NCME AIME-Con 2026 (October 5–7, 2026, Pittsburgh, PA); manuscript in preparation for Computers & Education
Anatomy of an automatically generated MAZE item A generated passage sentence with every seventh word replaced by three answer choices: one correct word and two distractors produced by a large language model. Student responses feed a scoring model that adjusts for word complexity and item difficulty. 1 · LLM-GENERATED, CURRICULUM-ALIGNED PASSAGE The little owl waited on the branch until the sky turned dark, and then she began to 2 · EVERY 7TH WORD BECOMES AN ITEM hunt correct table distractor quickly distractor Distractors come from three novel LLM methods plus a dictionary control. 3 · SCORING MODEL Student responses are normalized for word complexity and item difficulty, so scores stay comparable across every automatically generated form.
Figure 1. How an automated MAZE item is built. A large language model writes a curriculum-aligned passage; every seventh word is deleted and replaced with the correct word plus two distractors; a scoring model then adjusts for word complexity and item difficulty so that scores from different generated passages can be compared to one another.

Research focus

Making a century-old reading test work at classroom speed

The MAZE comprehension test has a long history in elementary education. It was popularized as part of the Curriculum-Based Measurement framework (Deno, 1985), the standardized testing process schools have used for decades to evaluate reading achievement, and it appears in the widely used DIBELS assessment. The format is simple: take a reading passage, delete every seventh word, and replace it with three choices: one correct word and two distractors. Many studies have provided validity evidence for standardized versions of the test.

The problem is the passages. Because traditional MAZE tests draw on a fixed bank of standardized passages, they are a poor fit for the intelligent tutoring systems that have been shown to substantially improve reading comprehension. Formative assessment requires frequent evaluation, and there simply are not enough standardized passages to go around. Conventional MAZE tests also cannot adapt to what a class is actually reading that week, which means the assessment can never reinforce the lesson it is measuring.

Our study addresses this by automatically creating MAZE tests from curriculum-aligned passages generated by large language models. The initial development phase is complete: we built four versions of an automated MAZE test that can be administered inside a reading app and scored automatically, and we collected a sample of real student responses. Three of the four use novel LLM-based methods to generate contextually appropriate distractors, and the fourth is a dictionary-based control, which lets us compare the approaches head to head.

My University Scholars year takes that from a working prototype to an instrument with evidence behind it. The work has three aims: develop scoring models that accurately predict student reading comprehension; examine reliability indices such as alternate-form reliability and internal consistency across automatically generated passages; and gather validity evidence, including correlations with established measures of reading comprehension. Success looks like a scoring method that accounts for word complexity and item difficulty well enough to normalize scores across generated passages, giving teachers a test they can use often without compromising its integrity. If the results instead show limitations, that is a finding worth having too, because it maps the boundaries of what intelligent tutoring systems can currently measure.

My responsibilities

What I actually do on this project

I joined Project MAZE in the fall of 2025 and helped develop the four versions of the automated MAZE test now under study, working on both the generation pipeline that produces passages and distractors and the delivery layer that administers the assessment inside our reading application and scores responses automatically. Because I am also the Lead Developer for the lab, I own that pipeline end to end: the prompts and post-processing that produce a usable passage, the data model that stores items and responses, and the production deployment that put the assessment in front of students to collect our first sample last fall.

For the University Scholars Program year, my primary responsibility is the scoring algorithm and the analyses that justify it. I am developing the model that converts raw responses into a comprehension score adjusted for word complexity and item difficulty, then running the reliability and validity analyses (alternate-form reliability, internal consistency, and correlations with established comprehension measures) that determine whether those scores mean what we claim they mean. A secondary line of work compares the three LLM distractor-generation strategies against the dictionary-based control to see which produces the most reliable and instructionally valid assessments, and how each affects student performance over time.

I meet weekly with Dr. Leite to review results and set next steps. I am a co-author on the paper accepted to NCME’s AIME-Con 2026, and we are preparing a manuscript based on this year’s results for Computers & Education.

← All research CV / Resume →

References

  • Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52(3), 219–232.
  • Shin, J., & McMaster, K. (2019). Relations between CBM (oral reading and maze) and reading comprehension on state achievement tests: A meta-analysis. Journal of School Psychology, 73, 131–149.
  • Xu, Z., et al. (2019). The effectiveness of intelligent tutoring systems on K–12 students’ reading comprehension: A meta-analysis. British Journal of Educational Technology, 50(6), 3119–3137.