In-class chat engagement significantly predicts higher test scores in MAIC courses¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Regression analysis found in-class chat message frequency and length significantly predict standardized test and final exam scores, controlling for Module 1 scores. → Jifan Yu 2024

Evidence¶

Jifan Yu 2024¶

Jifan Yu, Zheyuan Zhang, Daniel Zhang-li, Shangqing Tu, Zhanxin Hao, Ruimiao Li, Haoxuan Li, Yuanchun Wang, Hanming Li, Linlu Gong, Jie Cao, Jiayin Lin, Jinchang Zhou, Fei Qin, Haohua Wang, Jianxiao Jiang, Lijun Deng, Yisi Zhan, Chaojun Xiao, Xusheng Dai, Xuan Yan, Nianyi Lin, Nan Zhang, Ruixin Ni, Yang Dang, Lei Hou, Yu Zhang, Xu Han, Manli Li, Juanzi Li, Zhiyuan Liu, Huiqin Liu, Maosong Sun. (2024). From MOOC to MAIC: Reshaping Online Teaching and Learning through LLM-driven Agents. Preprint. Under review. https://arxiv.org/abs/2409.03512 doi:10.48550/arxiv.2409.03512

q2 · i? · associational · r2

Correlational analysis of over 100,000 learning records from the MAIC pilot, regressing standardized test scores on log message number and length with Module 1 controls (Table 2). The article reports 'in-class chat engagement is found to significantly predict higher test scores'; e.g., uncontrolled µ(log(MsgNum)) with AvgQuiz is 0.341***.

"A regression analysis of standardized test scores on in-class chat message metrics was performed, controlling for normalized scores from Module 1. As Table 2 shows, in-class chat engagement is found to significantly predict higher test scores."

Discussion¶

Related Claims¶

• Early-semester LMS interaction data predicts AI-assisted cheating risk in the final exam, with Logistic Regression achieving AUC = 0.763 under LOOCV — related
• Course-module views, assignment submissions, and video-access days are the most stable predictors, all lower in the high-risk group (Cliff's delta values from 0.50 to 0.60) — related
• Instruction type predicts post-test performance with a large effect size after controls (linear and quantile regression) — related
• Students in MAIC tend to prefer the passive continuous mode over interactive mode — related
• Pre-course SRL level statistically significantly predicted participation in collaborative chat activities — related
• Students' participation in preparatory exercises positively predicted their final exam grade in every intervention year (β = 0.16–0.31; R² = 0.03–0.16) — related
• More frequent exams are associated with better learning outcomes, including better grades and lower dropout, in classroom comparisons — related
• Behavioral-only metrics from an instrumented online textbook support early prediction of final course grades in flipped statistics courses — related