Explanation-quality distributions differed significantly between years for Physics 1A (2011 slightly lower) but not Physics 1B¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q3 · i1 small
Currency · newest study 2013 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2016: local conditions change quickly

Subclaims¶

q3 i? A chi-squared test found a statistically significant difference (p=0.022) between Physics 1A explanation-quality distributions for 2010 and 2011, with 2011 explanations slightly lower quality. → Bates 2013 q3 i? For Physics 1B, there was no statistically significant difference (p=0.66) between years. → Bates 2013 (2)

Evidence¶

Bates 2013¶

Bates, S. P., Galloway, R. K., Riise, J., and Homer, D. (2013). Assessing the quality of a student-generated question repository. https://arxiv.org/abs/1308.2202 doi:10.48550/arxiv.1308.2202

q3 · i1 · design · r2

Chi-squared test on explanation-category distributions for Physics 1A 2010 (N=150) and 2011 (N=200) samples, shown in Figure 3. The article reports "p=0.022" and that 2011 explanations "appear to be of slightly lower quality overall". No effect size is printed.

"A χ-squared test revealed a statistically significant difference between the distributions of the explanation categories for the Physics 1A data for 2010 and 2011 (shown in Figure 3) with p=0.022. The sampled explanations from the 2011 cohort appear to be of slightly lower quality overall compared to 2010 data."

Bates 2013 (2)¶

Bates, S. P., Galloway, R. K., Riise, J., and Homer, D. (2013). Assessing the quality of a student-generated question repository. https://arxiv.org/abs/1308.2202. doi:10.48550/arxiv.1308.2202

q3 · i? · design · r2

Chi-squared test on explanation-category distributions for Physics 1B 2010 (N=179) and 2011 (N=73) samples, shown in Figure 4. The article reports "no statistically significant difference (p=0.66)"; equivalence was not tested. No effect size is printed.

"Here, there is no statistically significant difference (p=0.66) between samples for two successive years, and the same overall pattern in the profile of explanation quality detailed above for the 1A questions is once again clearly apparent."

Discussion¶

Related Claims¶

• Question cognitive-level distributions differed significantly between years for Physics 1B but not Physics 1A — related
• 75% of student-authored physics questions met combined high-quality criteria (clear, correct, plausible distractors, above recall, adequate explanation) — related
• Over 95% of student-authored questions included an explanation, mostly of good or excellent quality — related
• The two textbook series differ significantly in the distribution of intelligences across learning objectives — related