Student DP performance differed significantly across the three course deployments, with the paper-based DT3 exam scoring lowest, likely due to administration mode¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2 · i2 medium
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? DP grades differed significantly across the three courses (F=15.519, p<0.001), with DT3 grades significantly lower than Touchalytics and Breached! (p<0.001). → Ahmad D. Suleiman 2026 q2 i2 Word count showed a strong positive correlation with DP grades (ρ=0.67, p<0.001), while time on task showed a negligible correlation (ρ=−0.05, p=0.76). → Ahmad D. Suleiman 2026 (2)

Evidence¶

Ahmad D. Suleiman 2026¶

Ahmad D. Suleiman, Daqing Hou, and Maliha Noushin Raida. (2026). LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning. arXiv preprint arXiv:2607.11032. https://arxiv.org/abs/2607.11032 doi:10.48550/arxiv.2607.11032

q2 · i? · design · r2

One-way ANOVA of Task A grades across the three courses found F=15.519, p<0.001; Tukey HSD showed DT3 (paper-based, N=40) significantly lower than the two web-based courses, which did not differ significantly.

"student DP performance differed significantly across the three deployments (𝐹= 15.519,𝑝< 0.001). Tukey HSD showed thatDT3 grades were significantly lower thanTouchalyticsandBreached! (𝑝< 0.001)"

Ahmad D. Suleiman 2026 (2)¶

Ahmad D. Suleiman, Daqing Hou, and Maliha Noushin Raida. (2026). LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning. arXiv preprint arXiv:2607.11032. https://arxiv.org/abs/2607.11032

q2 · i2 · design · r2

Spearman correlation analysis of performance data found word count strongly correlated with DP grades (ρ=0.67, p<0.001), while time on task correlated negligibly (ρ=−0.05, p=0.76), suggesting longer typed responses better articulated design justifications.

"word count showed a strong positive correlation with DP grades (𝜌= 0.67,𝑝< 0.001), suggesting that longer responses enabled students to better articulate and justify design choices."

Discussion¶

Related Claims¶

• Student performance on design problems correlates negligibly with traditional project grades, suggesting DPs capture distinct transfer-oriented HOT dimensions — related
• Keystroke patterns during web-based DP exams suggested planning, revision, and sustained cognitive engagement — related
• Requiring students to write out evaluation steps manually improved transfer to pen-and-paper exam performance between the two deployments — related
• Instructors value design problems for assessing higher-order thinking in PjBL but cite creation effort and scalability as adoption barriers — related