ChatGPT-generated HDR problems measure the same ability as human-created problems, with high internal consistency (Cronbach's α = 0.78) and no significant difficulty difference¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? ChatGPT-generated HDR problems and original human-created problems yielded high internal consistency (α = 0.78), suggesting both measure the same cognitive ability to detect misuse of knowledge. → Satoshi Takahashi 2026 q3 i? Score distributions for original and ChatGPT-generated problems were extremely similar, and Fisher's exact tests found no significant differences between problem pairs. → Satoshi Takahashi 2026 (2)

Evidence¶

Satoshi Takahashi 2026¶

Satoshi Takahashi, Atsushi Yoshikawa, Megumi Kose, Kenichi Suzuki, Chieko Inoue, Yumi Watanabe, and Mari Sawada. (2026). Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence. arXiv. https://arxiv.org/abs/2609.25790 doi:10.48550/arxiv.2609.25790

q3 · i? · design · r2

Experiment 1 with 100 working adults randomly assigned to two counterbalanced groups compared original and ChatGPT-generated HDR problems; Cronbach's α was computed for total scores, showing "a high internal consistency (α=0.78)".

"The original and ChatGPT-generated problems yielded a high internal consistency (α=0.78), suggesting that both measure the same cognitive ability to detect the misuse of knowledge."

Satoshi Takahashi 2026 (2)¶

Satoshi Takahashi, Atsushi Yoshikawa, Megumi Kose, Kenichi Suzuki, Chieko Inoue, Yumi Watanabe, and Mari Sawada. (2026). Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence. arXiv. https://arxiv.org/abs/2609.25790

q3 · i? · design · r2

Experiment 1 score-distribution analysis: mean scores and 95% confidence intervals for original vs. ChatGPT-generated problems showed "extremely similar" trends, and Fisher's exact tests found no significant differences.

"Figure 1 compares the distribution of the descriptive scores for the original and ChatGPT-generated problems. The trends are extremely similar, indicating no difference in difficulty."

Discussion¶

Related Claims¶

• Automated HDR problem generation makes repeated and longitudinal assessment feasible by producing structurally equivalent cases with different contexts — related
• HDR scores correlate only weakly with reading comprehension (α = 0.36 and 0.41), indicating the task measures practical application ability rather than reading skill — related
• Both proposed adjustment methods (IEA and BA) reduce RMSE of item difficulty estimates from direct 2PL-IRT fitting, with BA yielding more consistent (lower-variance) estimates in simulation — related
• Students did not consistently detect scientific errors in ChatGPT responses, and students with stronger prior knowledge evaluated AI responses more critically — related