Keystroke-feature regression models fail to predict final writing grade at any point during the writing process¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2022 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? None of the regression models (random forest, SVM, naive Bayes) at any time- or keystroke-based timepoint outperformed the mean final grade baseline for predicting final grade. → Rianne Conijn 2022

Evidence¶

Rianne Conijn 2022¶

Rianne Conijn, Christine Cook, Menno van Zaanen, Luuk Van Waes. (2022). Early prediction of writing quality using keystroke logging. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-021-00268-w

q2 · i? · associational · r2

Regression results (RQ2): random forest, support vector machine, and naive Bayes models trained on windowed and cumulative keystroke feature sets at twelve timepoints, evaluated with RMSE against the mean-grade baseline. No effect size printed; the null indicates no detected predictive gain, not equivalence.

"None of the models outperformed the baseline (mean final grade) at any timepoint in the writing process. Thus, the keystroke features cannot be used for the early prediction of final grade, nor for the prediction of final grade once the writing process is finished."

Discussion¶

Related Claims¶

• Which keystroke features correlate with writing grade changes across stages of the writing process: pause-variance and focus-shift features matter early, sentence-transition and event features later — reports the opposite
• Binary classification of students who might need support from keystroke data only slightly exceeds the majority-class baseline (highest AUC 0.57) — related
• Over the full writing process, only one of 54 keystroke features (percentage of long pauses between words) correlates significantly with final writing grade in ESL academic summaries — reports the opposite
• On real tutor log data, the models beat mean-time prediction but most of the gain is captured by a simple baseline, and the learning extension slightly overfits — related
• The proposed model's effectiveness was demonstrated by applying it to actual keystroke log datasets — related