Use AI-driven item generation to create discriminating questions at a fraction of the cost and offload easier questions to an AI¶

Strategy · All strategies
Evidence · no claims cited

Description¶

Based on the field-study IRT results, the authors recommend that instructors and testing platforms use AI-driven item generation—specifically their iterative refinement procedure—to produce questions with reliable discriminatory power at low cost, and to delegate creation of easier questions to an AI. The recommendation rests on the finding that AI items matched or exceeded the discrimination of standardized-test items while being somewhat easier, making them well suited for routine, calibrated classroom assessment.

Design Implications¶

Context¶

Requirements¶

• Instructor-provided course materials to tailor questions to course content
• An iterative generate–judge–refine loop with an AI judge applying appropriateness criteria

Constraints¶

• The comparison to standardized questions was evaluated only for statistics courses, limiting generalizability
• Evaluation was limited to short 10-question multiple-choice tests

Target Learners¶

• college instructors and testing platforms serving college students

Target Learning Goals¶

• efficient creation of psychometrically adequate course assessments

Affordances¶

• Iterative Llm Refinement Item Generation

Related Strategies¶

• Use an SLM for rapid large-scale question generation with an LLM providing validation feedback

Examples¶

-

Key Sources¶

• Calvin Isley, Joshua Gilbert, Evangelos Kassos, Michaela Kocher, Allen Nie, Emma Brunskill, Ben Domingue, Jake Hofman, Joscha Legewie, Teddy Svoronos, Charlotte Tuminelli, Sharad Goel. (2025). Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study. https://arxiv.org/abs/2508.08314