Overlapping concepts (sharing post-test questions with other concepts) yield higher student scores and higher model R², and multi-task modeling across all concepts is detrimental for these concepts¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2023: tool and model accuracy changes quickly

Subclaims¶

q2 i? Students scored significantly higher on overlapping concepts (0.718) than non-overlapping concepts (0.550), and hybrid framework models yielded R² of 0.339 (overlapping) versus 0.157 (non-overlapping). → Henderson 2020 q2 i? Optimal models for overlapping concepts were primarily single-task, indicating that multi-task modeling across all concepts including irrelevant ones is detrimental for overlapping concepts. → Henderson 2020 (2)

Evidence¶

Henderson 2020¶

Henderson, N., Kumaran, V., Min, W., Mott, B., Wu, Z., Boulden, D., Lord, T., Reichsman, F., Dorsey, C., Wiebe, E., & Lester, J. (2020). Enhancing Student Competency Models for Game-Based Learning with a Hybrid Stealth Assessment Framework. Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020), 92-103. https://educationaldatamining.org/EDM2020/proceedings/

q2 · i? · associational · r2

Analysis of 316 students' post-test scores and model performance by concept overlap status in Geniventure. Overlapping concepts achieved "average scores of 0.718 and 0.550" and R² of "0.339 (overlapping) and 0.157 (non-overlapping)."

"Student scores were significantly higher for overlapping concepts as opposed to non -overlapping questions, achieving average scores of 0.718 and 0.550, respectively. This trend is also present in our competency models, as the optimal models in our hybrid framework yield R2 values of 0.339 (overlapping) and 0.157 (non-overlapping) on average."

Henderson 2020 (2)¶

Henderson, N., Kumaran, V., Min, W., Mott, B., Wu, Z., Boulden, D., Lord, T., Reichsman, F., Dorsey, C., Wiebe, E., & Lester, J. (2020). Enhancing Student Competency Models for Game-Based Learning with a Hybrid Stealth Assessment Framework. Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020), 92-103. https://educationaldatamining.org/EDM2020/proceedings/

q2 · i? · associational · r2

Model selection analysis for overlapping concepts in the hybrid framework. The authors found optimal models were "primarily single-task models" and conclude multi-task modeling "is actually detrimental" for these concepts.

"the optimal models for the overlapping concepts were primarily single-task models, with the exception of one model. This indicates that multi-task modeling across all the concepts including both relevant and irrelevant concepts is actually detrimental in terms of achieving higher predictive performance."

Discussion¶

Related Claims¶

• GRU is the lowest-performing model across both single-task and multi-task competency modeling, likely due to its two gating units versus LSTMs' three
• The single-task Random Forest achieves the best average predictive performance but is optimal for only 7 of 15 concepts
• A hybrid stealth assessment framework selecting the optimal model per concept outperforms any single homogeneous modeling technique for predicting student competencies
• Sequential deep learning models outperform static models for concepts mapped to multiple post-test questions — related
• Concepts associated with only a single post-test question yield markedly low predictive performance across all modeling approaches — related