Cognitive artifact scoring in the digital workbook achieved high intercoder reliability, with average Spearman's rho of 0.97 overall¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Open-ended task scoring by instructed student workers showed substantial intercoder reliability assessed on approximately 10% of the data. → Kubsch 2025

Evidence¶

Kubsch 2025¶

Kubsch, M., Strauß, S., Grimm, A., Gombert, S., Drachsler, H., Neumann, K., & Rummel, N. (2025). Self-regulated Learning in the Digitally Enhanced Science Classroom: Toward an Early Warning System. Educational Psychology Review. https://doi.org/10.1007/s10648-025-10011-9

q2 · i? · associational · r2

Intercoder reliability assessment on a randomly selected ~10% subsample of open-ended task responses, independently coded by two coders, with drift checks via regular reviews of coded data.

"An average Spearman's rho of 0.97 with a minimum of 0.68 indicated substantial intercoder reliability. Across the different scores we found an average Spearman's rho of 0.94 for knowledge elements, an average Spearman's rho of 0.91 for engagement in scientific practices, and an average Spearman's rho of 0.94 for learning performances."

Discussion¶

Related Claims¶

• Within-judge variance of the repeated LLM-judge rubric is negligible, but cross-model-family agreement remains untested — related
• In an AI-based early warning system, the predictive importance of CAM dimensions shifted over an instructional unit, with affect/metacognition self-reports leading early and cognitive artifacts gaining accuracy toward the end — related