NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward¶

Claim · All claims
Evidence · 1 study (2 entries) · quant-synthesis r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

Reading the evidence codes

q is the design tier, i the impact, n the sample, and the kind with its r the rigour judged within that kind: what each code means.

This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.

q3 i? Across reviewed studies reporting metrics, mean performance reached 0.54 in kappa and 0.79 in accuracy. → Rafael Ferreira Mello 2024 q3 i? From 2021 onward, deep learning models (LSTM, DNN, BERT) outperformed others in the majority of recent studies. → Rafael Ferreira Mello 2024 (2)

Evidence¶

Rafael Ferreira Mello 2024¶

Rafael Ferreira Mello, Elyda Freitas, Luciano Cabral, Filipe Dwan Pereira, Luiz Rodrigues, Mladen Rakovic, Jackson Raniel, Dragan Gašević. (2024). Words of Wisdom: A Journey through the Realm of Natural Language Processing for Learning Analytics—A Systematic Literature Review. Journal of Learning Analytics, 11(3), 82–105. https://doi.org/10.18608/jla.2024.8403

q3 · i? · quant-synthesis · r2

Synthesis of Cohen's kappa and accuracy across reviewed studies (Table 4, RQ7): overall kappa mean 0.54 (SD 0.19, 28 papers) and accuracy mean 0.79 (SD 0.10, 45 papers). The review notes a kappa of 0.54 denotes moderate agreement and 0.79 accuracy a strong indicator of good performance.

"However, the mean performance metrics observed in the field significantly exceed these thresholds, typically reaching 0.54 in kappa and 0.79 in accuracy."

Rafael Ferreira Mello 2024 (2)¶

q3 · i? · quant-synthesis · r2

Year-by-year analysis of the best-performing algorithm per study (Figure 8, RQ7). Random Forest predominated in earlier years, peaking in 2020; from 2021 onward deep learning models led in the majority of recent studies. The review cautions that dataset and other factors influence outcomes, so results are not a definitive benchmark.

"In contrast, if we focus exclusively on research from 2021 onward, it becomes evident that deep learning models, namely LSTM, DNN (Deep Neural Networks) (Canale et al., 2021), and BERT, have outperformed others in the majority of recent studies."

Discussion¶

The fine-tuned BERT model generalizes to an external dataset collected by a different research team, maintaining almost perfect agreement with human coders — a narrower finding that bears on this claim
Random Forest is the most commonly used NLP-supporting algorithm in LA, while deep learning models (LSTM, BERT) have been widely adopted since 2021 — related
Collaborative learning is the most prominent educational task addressed with NLP in learning analytics research, followed by assessment and feedback — related
NLP-for-LA research is heavily biased toward English-language texts and small datasets, with almost no open datasets — related
Most NLP-for-LA studies (89.10%) never apply their models in real educational settings, revealing a gap between model development and practical use — related
A support vector machine classifier outperformed decision tree and random forest models in predicting cognitive engagement levels of online discussion posts — related
A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
The review's evidence base comprises 42 peer-reviewed publications selected from 922 records with high inter-rater agreement (Cohen's kappa = 0.88) — related
Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — a narrower finding that bears on this claim
An independent human second coder reached moderate agreement (kappa = 0.395) with the AI-assisted primary coding on the five-category outcome — a narrower finding that bears on this claim
Transformer-based NLP models achieve automated essay scoring with correlations to human grading exceeding 0.85 — a narrower finding that bears on this claim
Benchmarking shows NLP-based automated essay scoring achieves grading times up to 80% faster than human graders while maintaining comparable accuracy — related
NLP-based feedback generation for textual student answers in large courses reduced grading effort by up to 85% with high precision (reported from Bernius et al.) — related
An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
Essays within each tool group showed consistent homogeneity across named entities, n-grams, and topic ontology — related

Page metadata

Field	Value
`type`	`claim`
`id`	`nlp-la-performance-kappa-accuracy-benchmarks`
`title`	`NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward`
`description`	`NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward`
`status`	`review`
`evidence_strength`	`moderate`
`newest_evidence`	`2024`
`recheck_by`	`2027`
`generated`	by `process:wiki-ingest` on `2026-10-09`
`trust`	unverified — no one has confirmed this page's content
`sources`	2 — `rafael-ferreira-mello-2024`, `rafael-ferreira-mello-2024-2`

Frontmatter is the page's machine-readable record — see the Schema & Guide.