Under unseen-answers evaluation, the German Transformer model's F-Scores are numerically similar to state-of-the-art Transformer results for English¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In stratified unseen-answers evaluation, German F-Scores (81.7–86.2) are numerically similar to English SemEval-2013 corpus results (83.4, 91.2). → Ulrike Padó 2024

Evidence¶

Ulrike Padó 2024¶

Ulrike Padó, Yunus Eryilmaz, Larissa Kirschner. (2024). Short-Answer Grading for German: Addressing the Challenges. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-023-00383-w

q2 · i? · design · r2

Cross-language comparison using a fixed train-test split stratified by question (unseen answers), an easier task than unseen questions. German F-Scores range from 81.7 (CREG) to 86.2 (ASAP-DE, SAF-DE); English albert-large results are 83.4 (SciEntsBank) and 91.2 (Beetle). The authors state this "indicates" rather than proves comparability.

"Table 5 does demonstrate, though, that evaluation results for our German Transformer model in terms of F-Score are numerically similar to state-of-the-art Transformer results for the SemEval-2013 English standard corpora"

Discussion¶

Related Claims¶

• A fine-tuned SBERT Transformer SAG model for German approximates but does not beat hand-crafted feature-based models in unseen-question evaluation — related
• SAG model prediction performance varies widely across corpora even with the same architecture, so benchmark results cannot predict performance on new data sets — related
• Fine-tuned LaBSE achieves about 86.5% accuracy classifying all answers as correct or incorrect on a large multilingual dataset — related
• The German Transformer model's tokenization handles native-speaker morphology well but struggles with learner-language errors, possibly explaining lower CREG results — related