Where labelled training data is limited, adding hand-crafted features improves the performance of neural text evaluation models¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Nadeem et al. (2019) found that particularly where labelled training data is limited, adding hand-crafted features improves neural model performance. → Xiaoyu Bai · Manfred Stede 2023

Evidence¶

Xiaoyu Bai · Manfred Stede 2023¶

Xiaoyu Bai · Manfred Stede. (2023). A Survey of Current Machine Learning Approaches to Student Free-Text Evaluation for Intelligent Tutoring. International Journal of Artificial Intelligence in Education (2023) 33:994–1032. https://doi.org/10.1007/s40593-022-00323-0

q2 · i? · review · r2

The survey reports Nadeem et al. (2019)'s finding in the context of hybrid neural and feature-based approaches to essay scoring; Nadeem et al. also used contextualised word embeddings from a static BERT model to initialise their LSTM-based architecture. No effect size is printed.

"Nadeem et al. ( 2019) found that particularly where labelled training data is limited, adding hand-crafted features improve the performance of neural models."

Discussion¶

Related Claims¶

• Feature-based short-answer scoring systems may be less susceptible to certain student gaming attempts than end-to-end neural models — related
• Holistic scoring of German university students' essays posed significant challenges to both a neural model and an SVM with n-gram features — related
• Concatenating hand-crafted linguistic features with neural essay representations significantly increases essay scoring performance — related
• A fine-tuned SBERT Transformer SAG model for German approximates but does not beat hand-crafted feature-based models in unseen-question evaluation — related