A Bayesian student model's diagnoses agree highly with expert averages on a written exam but poorly on the computerized test¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2013 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The BSM applied to the written exam showed a high degree of agreement with the experts' average diagnosis, while the BSM applied to the computer test showed a low degree of agreement. → Millán-Valldeperas 2013

Evidence¶

Millán-Valldeperas 2013¶

Millán-Valldeperas, E.; Descalço, L.; Castillo, G.; Oliveira, P.; Diogo, S. (2013). Using Bayesian networks to improve knowledge assessment. Elsevier. http://hdl.handle.net/10630/7068

q2 · i? · design · r2

Evaluation study in which 152 students took both a PMatE computerized test and a written exam graded by three experts; Bayesian student models estimated knowledge on the same 12 concepts and "inter-rater agreement" was computed across measures. The article reports a high degree of agreement for the written-exam BSM and a low degree for the computer-test BSM; no effect size is printed.

"Results show a high degree of agreement among the scores given by the experts and also among the diagnosis provided by the BSM in the written exam and expert’s average, but a low degree of agreement among the diagnosis provided by the BSM in the computer test and expert’s average"

Discussion¶

Related Claims¶

• Diagnosis capability of domain-specific Bayesian student models was tested with 152 students across 12 first-degree-equation concepts — related
• Expert graders show high agreement among themselves when scoring a written exam on first degree equations — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• The generic Bayesian student model had been evaluated with simulated students before its first real application — related