Multiple-choice single-select items are less difficult than open-ended response questions, and delayed perceived task difficulty ratings relate more strongly to relative item difficulty for open-ended items¶

Claim · All claims
Evidence · 1 study (3 entries) · associational r2 · q2 · i1–i3
Currency · newest study 2025 · checked against its sources 2026-10-20 (automated, 3 of 3 entries) · not reviewed by a person

Subclaims¶

q2 i1 Relative item difficulty is significantly lower for multiple-choice single-select items than for open-ended response questions (ΔM = 0.49, d = 0.38). → Schuessler 2025 q2 i3 The correlation between perceived task difficulty and relative item difficulty is higher for open-ended response questions (r = -0.80) than for multiple-choice items (r = -0.45), a large difference (Cohen's q = 0.62). → Schuessler 2025 (2) q2 i? Relative item difficulty of open-ended items explains 64.3% of variance in perceived task difficulty versus 19.9% for multiple-choice items. → Schuessler 2025 (3)

Evidence¶

Schuessler 2025¶

Schuessler, K., Fischer, V., & Walpuski, M. (2025). Investigating construct validity of cognitive load measurement using single-item subjective rating scales. Instructional Science. https://doi.org/10.1007/s11251-024-09692-6

q2 · i1 · causal · r1

Item-level comparison of 61 multiple-choice single-select items and 65 open-ended items from the competence test, using a bootstrapped t-test with adjusted results. The article reports the adjusted difference as "significant, t(112.133) = 2.13, p = 0.035, d = 0.38".

"This difference, ΔM = 0.49, 95% BCa CI [0.03, 0.94], is significant, t(112.133) = 2.13, p = 0.035, d = 0.38 (as the Levene–test revealed significant variance differences we report the adjusted results)."

Schuessler 2025 (2)¶

Schuessler, K., Fischer, V., & Walpuski, M. (2025). Investigating construct validity of cognitive load measurement using single-item subjective rating scales. Instructional Science. https://doi.org/10.1007/s11251-024-09692-6

q2 · i3 · associational · r2

Comparison of the open-ended correlation (r = -0.80, nitems open = 65) against the multiple-choice correlation (r = -0.45, nitems MC = 61) with perceived task difficulty; the comparison of correlations is computed for independent samples and yields the printed effect size "a large effect (q = 0.62, z = 3.41, p ≤ 0.001)".

"For the difference between correlations Cohen's q reveals a large effect (q = 0.62, z = 3.41, p ≤ 0.001)."

Schuessler 2025 (3)¶

Schuessler, K., Fischer, V., & Walpuski, M. (2025). Investigating construct validity of cognitive load measurement using single-item subjective rating scales. Instructional Science. https://doi.org/10.1007/s11251-024-09692-6

q2 · i? · associational · r2

Regressions predicting perceived task difficulty by relative item difficulty separately per answer format; the open-ended model explains 64.3% of variance, which the article calls "clearly higher than 19.9%" for the multiple-choice model. Only R² values are printed, so no standardized effect size applies.

"This is clearly higher than 19.9% of variance in perceived task difficulty, which is explained when using the relative item difficulty of multiple-choice single-select items as a predictor, R 2 = 0.20, F(1, 59) = 14.70, p ≤ 0.001, nitems MC = 61."

Discussion¶

Related Claims¶

• Participants rated the complex three-population scenario as more difficult than the simple one-population scenario — related
• Items requiring integration have higher relative item difficulty than items requiring selecting or organizing, and delayed perceived task difficulty ratings appear closer to integrating-item difficulty, though that correlation comparison is not significant
• Delayed perceived task difficulty ratings relate more strongly to the relative difficulty of the more demanding items within a task set, consistent with delayed ratings being affected by the most demanding item — related
• Perceived task difficulty ratings show a strong relation to Rasch-estimated relative task difficulty, supporting construct validity of the single-item scale — a broader claim this one bears on
• Invested mental effort and perceived task difficulty are strongly correlated yet relate differently to relative task difficulty, indicating the two ratings measure different aspects of overall cognitive load — related
• Re-analyses support PSVT:R structural validity: degree of rotation explains 66.9% of item-difficulty variance, though modified problematic items show no rotation-difficulty relationship — related