Feature operationalization can reverse the direction of the relationship with essay score, as with document length and TTR variants¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2016 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In the FCE model document length carried a negative weight while sentence length was positive, and TTR and corrected TTR appeared on opposite sides of the weight table. → Sowmya Vajjala 2016

Evidence¶

Sowmya Vajjala 2016¶

Sowmya Vajjala. (2016). Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education. https://arxiv.org/abs/1612.00729

q2 · i? · associational · r2

Analysis of regression feature weights (Table 12) for the FCE model: document length is a negative-valued predictor while sentence length is positive, reversing the TOEFL11 pattern where document length had a high positive weight.

"An interesting aspect of this list of features is that, docum ent length is a negative valued predictor, while sentence length is a positive predictor for FCE texts."

Sowmya Vajjala 2016 (2)¶

Sowmya Vajjala. (2016). Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education. https://arxiv.org/abs/1612.00729

q2 · i? · associational · r2

The authors conclude from the FCE weight table (e.g., TTR negative, corrected TTR positive) that "the exact operationalization of t he features can change the direction of the relationship with the score variable".

"Thus, the exact operationalization of t he features can change the direction of the relationship with the score variable."

Discussion¶

Related Claims¶

• The most predictive linguistic features differ across datasets, overlapping only in lexical diversity measures — related
• Document length alone carries some signal (word-count classifier ROC-AUC 0.68), but the fixed-word sliding-window design largely prevents stylometric features from capturing length differences — related
• Type-token ratio showed a non-monotonic relationship with predicted difficulty, with both very low and very high lexical diversity increasing predicted text difficulty — related
• Systematic linguistic differences exist between demographic groups in PISA text responses, including response length, lexical diversity, and out-of-vocabulary tokens — related