Machine translation of training and test data does not degrade monolingual content scoring performance and can normalize learner errors¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In Study 1, scoring machine-translated datasets performed in the same ballpark as scoring the originals, sometimes slightly higher for Chinese as the source language. → Andrea Horbach 2024 q1 i? Manual probing found misspellings such as vinagar often translated correctly, supporting a normalization-by-translation effect. → Andrea Horbach 2024

Evidence¶

Andrea Horbach 2024¶

Andrea Horbach, Joey Pehlke, Ronja Laarmann-Quante, Yuning Ding. (2024). Crosslingual Content Scoring in Five Languages Using Machine-Translation and Multilingual Transformer Models. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-023-00370-1

q2 · i? · design · r2

Study 1 monolingual baseline experiments translating both training and test data with DeepL into each other language. The article reports results "in the same ballpark as the original data" and attributes this to normalization of spelling errors, with manual checks of misspelled sentences.

"In most cases, results produced on translated datasets are in the same ballpark as the original data, sometimes even slightly higher especially in the case of Chinese as the original language. This points at a normalizing effect that machine translation can have, for example on spelling errors."

Discussion¶

Related Claims¶

• Crosslingual content scoring transfer works better between closely related languages, with Chinese the hardest target in this setup — related
• M-BERT with machine translation is the best crosslingual method, outperforming logistic regression with MT and M-BERT zero-shot — related
• Monolingual English BERT shows only a slight advantage over multilingual BERT for English content scoring — related
• Higher-scored learner answers tend to be longer, and crowdsourced answers tend to be shorter than original classroom data — related
• Part of the crosslingual performance drop is attributable to differences between learner populations and datasets, not only language — related