Machine learning detects five of eight reflective-writing categories with substantial or almost perfect reliability and three with moderate reliability (Cohen's κ .53–.85)¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2019 · checked against its sources 2026-10-27 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q3 i? In a standardized evaluation, five out of eight categories of reflective writing were detected automatically with substantial or almost perfect reliability, while the other three were detected with moderate reliability (Cohen's κ ranges between .53 and .85). → Thomas Daniel Ullmann 2019

Evidence¶

Thomas Daniel Ullmann 2019¶

Thomas Daniel Ullmann. (2019). Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. International Journal of Artificial Intelligence in Education (2019) 29:217–257. https://doi.org/10.1007/s40593-019-00174-2

q3 · i? · design · r2

Evaluation of machine learning models trained on a random 80% of labeled sentences and tested on the held-out 20%, in a standardized process applied to all eight category data sets; the article reports "Cohen’s κ ranges between .53 and .85" across categories.

"Overall, the standardized evaluation shows that five out of eight categories can be detected automatically with substantial or almost perfect reliability, while the other three categories can be detected with moderate reliability (Cohen’s κ ranges between .53 and .85)."

Discussion¶

Related Claims¶

• The eight-category model of reflective writing can be reliably annotated and showed empirical validity in this context — possibly the same claim (merge candidate)
• NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — related
• A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
• Automatic DAC coding of dialogue acts agrees substantially with manual coding (79.0% agreement, kappa 0.75) — related
• The rule-based recipe annotation system reaches near-perfect inter-annotator agreement (Cohen's Kappa 0.85) and average accuracy 0.95 in a validation on sampled recipe segments. — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• A structured negotiated-coding process reached nearly 98% interrater agreement before machine learning — related
• Prior machine learning research on reflection detection covers only partial models with no evidence of empirical validity and mostly no reported inter-rater reliability — related