Ablation tests identify unique trait-relevant information in specific feature groups: readability and syntactic complexity for content, cohesion and readability for organization, and occurrence, length, or error features for language¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Removing readability and syntactic complexity features dropped content-trait performance across all four prompts, indicating information the embeddings and other features could not capture. → Lohmann 2025 q2 i? Length features emerged as an essential feature type across traits, capturing unique text characteristics. → Lohmann 2025 (2)

Evidence¶

Lohmann 2025¶

Lohmann, J. F., Junge, F., Möller, J., Fleckenstein, J., Trüb, R., Keller, S., Jansen, T., & Horbach, A. (2025). Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners' Argumentative Essays. International Journal of Artificial Intelligence in Education, 35, 1178–1217. https://doi.org/10.1007/s40593-024-00426-w

q2 · i? · design · r2

Ablation tests iteratively removed one of nine feature groups from the full hybrid model, rerunning each trait- and prompt-specific model; Figure 5 shows the performance drops.

"the performance of models for the content traits dropped across all four prompts when readability and syntactic complexity features were removed."

Lohmann 2025 (2)¶

Lohmann, J. F., Junge, F., Möller, J., Fleckenstein, J., Trüb, R., Keller, S., Jansen, T., & Horbach, A. (2025). Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners' Argumentative Essays. International Journal of Artificial Intelligence in Education, 35, 1178–1217. https://doi.org/10.1007/s40593-024-00426-w

q2 · i? · design · r2

Same ablation series on the hybrid models; consistent drops across prompts indicated a feature type contains trait-relevant information not captured by embeddings or other features.

"Throughout traits, length features, in particular, emerged as an essential feature type capturing important and unique text characteristics for judging the student essays."

Discussion¶

Related Claims¶

• Addition tests show trait-specific feature groups that most consistently enhance embedding-based models: morphological complexity for content, length for organization, and lexical sophistication, error, and occurrence features for language — related
• A response-length-only baseline scores poorly, showing scores depend on broader linguistic features — related
• For trait-specific automated essay scoring trained on a single resource, feature-based DNNs slightly outperform embedding-based DNNs, winning 11 of 16 trait-prompt comparisons — related
• Adding fluency features to content features sharply increases scoring QWK for every prompt — related
• Ablation study: removing any component lowers evaluation AUC, and removing all additional features yields the lowest public and private AUCs — related
• Removing grammar and vocabulary features drops scoring QWK by 7-13% across prompts — related
• Complex DNN architectures offer little advantage over simple linear ridge-regression baselines for trait scoring, implying nonlinearities and interactions were of minor importance — reports the opposite