Automated HDR problem generation makes repeated and longitudinal assessment feasible by producing structurally equivalent cases with different contexts¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? ChatGPT can generate multiple cases that are structurally equivalent but contextually different, enabling evaluation while suppressing item bias in re-measurement and making repeated and longitudinal assessments feasible. → Satoshi Takahashi 2026

Evidence¶

Satoshi Takahashi 2026¶

Satoshi Takahashi, Atsushi Yoshikawa, Megumi Kose, Kenichi Suzuki, Chieko Inoue, Yumi Watanabe, and Mari Sawada. (2026). Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence. arXiv. https://arxiv.org/abs/2609.25790 doi:10.48550/arxiv.2609.25790

q2 · i? · design · r2

Authors' discussion (Section 6.3) interpreting their Experiment 1 reproducibility results: traditional reuse of the same problem leads to memory dependence and answer reuse, making continuous evaluation of genuine ability difficult.

"ChatGPT, as employed in this study, can generate multiple cases that are structurally equivalent but contextually different, enabling evaluation while suppressing item bias, even in re-measurement."

Discussion¶

Related Claims¶

• ChatGPT-generated HDR problems measure the same ability as human-created problems, with high internal consistency (Cronbach's α = 0.78) and no significant difficulty difference — related
• Human scoring and feedback for HDR takes about 11 minutes 30 seconds per learner, while GPT-based scoring and feedback is nearly instant — related
• Benchmark success on human-designed assessments may not establish that the underlying abilities are being measured — related
• HDR scores correlate only weakly with reading comprehension (α = 0.36 and 0.41), indicating the task measures practical application ability rather than reading skill — related
• Hint-based prompts raise ChatGPT scoring agreement with human experts to 98–100%, while no-hint prompts are unstable (Q2 precision 0.42) — related