Students rated DPs positively, did not distinguish the two LLMs, and disagreed with experts on the same DPs, rating feasibility significantly lower¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2 · i1 small
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Cognitive Complexity received the highest student mean rating (M=4.35, 87% agreeing) and Scenario Quality was rated highly (M=4.21), while Feasibility was lowest (M=3.63). → Ahmad D. Suleiman 2026 q2 i1 Student and expert ratings on the 39 shared DPs showed no agreement overall (κ=−0.11); students rated Scenario Quality higher than experts (p=0.049, d=0.33) but Feasibility lower (p<0.001, d=0.74). → Ahmad D. Suleiman 2026 (2)

Evidence¶

Ahmad D. Suleiman 2026¶

Ahmad D. Suleiman, Daqing Hou, and Maliha Noushin Raida. (2026). LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning. arXiv preprint arXiv:2607.11032. https://arxiv.org/abs/2607.11032 doi:10.48550/arxiv.2607.11032

q2 · i? · design · r2

Descriptive statistics of Task B student ratings on 5-point Likert scales in three course deployments showed Cognitive Complexity highest at M=4.35 (87% agreement) and Feasibility lowest at M=3.63, with 14% disagreeing.

"Cognitive Complexityreceived the highest mean rating (𝑀= 4.35), with 87% agreeing that the prompts engaged HOT.Scenario Qualitywas also rated highly ( 𝑀= 4.21), with 88% finding the scenarios realistic and meaningful."

Ahmad D. Suleiman 2026 (2)¶

Ahmad D. Suleiman, Daqing Hou, and Maliha Noushin Raida. (2026). LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning. arXiv preprint arXiv:2607.11032. https://arxiv.org/abs/2607.11032

q2 · i1 · design · r2

Weighted Cohen's κ and paired t-tests comparing student and expert ratings on 39 shared DPs found "no agreement overall (𝜅=− 0.11)"; students rated Scenario Quality higher (d=0.33) and Feasibility lower (d=0.74) than experts.

"Comparing student and expert ratings on the 39 shared DPs, there was no agreement overall (𝜅=− 0.11), indicating that the two groups often judged the same DPs differently."

Discussion¶

Related Claims¶

• A reasoning-oriented LLM outperformed a standard LLM on scenario quality and feasibility of generated design problems, with no differences on alignment, cognitive complexity, or clarity — related
• Half of LLM-generated design problems received perfect expert ratings, though common failure modes included scenarios too similar to the original project and ambiguous scope — related
• Instructors value design problems for assessing higher-order thinking in PjBL but cite creation effort and scalability as adoption barriers — related
• Undergraduate learners rated CyberAGENTS positively, with scenario authenticity highest and challenge difficulty alignment lowest — related