Prior machine learning research on reflection detection covers only partial models with no evidence of empirical validity and mostly no reported inter-rater reliability¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Existing ML studies of reflection detection use less comprehensive models that lack evidence of empirical validity, and apart from two papers did not report inter-rater reliability values. → Thomas Daniel Ullmann 2019

Evidence¶

Thomas Daniel Ullmann 2019¶

Thomas Daniel Ullmann. (2019). Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. International Journal of Artificial Intelligence in Education (2019) 29:217–257. https://doi.org/10.1007/s40593-019-00174-2

q2 · i? · design · r2

Narrative literature review of ML-based approaches (e.g. Yusuf 2011, Cheng 2017, Liu et al. 2018, Kovanović et al. 2018); the review reports the models "show any evidence for empirical validity" was missing and that apart from two papers inter-rater reliability was not reported.

"Furthermore, the review shows that none of the models show any evidence for empirical validity. While many of the mentioned categories can be traced back to the theory or reflective thinking and thus have face validity, evidence of empirical validity would have strengthened the case"

Discussion¶

Related Claims¶

• Machine learning detects five of eight reflective-writing categories with substantial or almost perfect reliability and three with moderate reliability (Cohen's κ .53–.85) — related
• The eight-category model of reflective writing can be reliably annotated and showed empirical validity in this context — related