Use LLM-generated design problems in supervised, time-bound exam-like settings to assess individual higher-order thinking where AI use or teamwork obscures mastery¶

Strategy · All strategies
Evidence · 1 claim (1 mixed) · 1 study (1 design), q2 · 1 of 1 report an effect size · 1 claim rests on one study

Description¶

The article recommends deploying DPs as short, supervised, individual assessments to elicit individual reasoning in novel contexts, particularly when generative AI use or collaborative work may compromise assessment validity. Instructors noted DPs are "most convincing in settings requiring students to work independently, such as time-bound in-class assessments." LLMs reduce the creation bottleneck while instructors remain responsible for review and adaptation of generated prompts.

Design Implications¶

Context¶

Requirements¶

• DPs tailored to specific project learning objectives; consistent grading rubric; instructor review and adaptation of LLM-generated prompts

Constraints¶

• Paper-based administration may constrain response length and depress grades; open-ended grading is subjective and may require additional support to scale

Target Learners¶

• undergraduate computing students in project-based courses

Target Learning Goals¶

• individual demonstration of transfer-oriented higher-order thinking

Claims¶

• Student DP performance differed significantly across the three course deployments, with the paper-based DT3 exam scoring lowest, likely due to administration mode [~W]

Related Strategies¶

• Redesign assessment reflectively for possible AI use, using supervised tasks for lower-order skills and intentional AI use with reflection for higher-order work

Examples¶

-

Key Sources¶

• Ahmad D. Suleiman, Daqing Hou, and Maliha Noushin Raida. (2026). LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning. arXiv preprint arXiv:2607.11032. https://arxiv.org/abs/2607.11032