SAG model prediction performance varies widely across corpora even with the same architecture, so benchmark results cannot predict performance on new data sets¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? For the same model architecture and language model, prediction performance varies widely across German corpora (F-Score 59.8 to 72.1 in unseen-question evaluation). → Ulrike Padó 2024

Evidence¶

Ulrike Padó 2024¶

Ulrike Padó, Yunus Eryilmaz, Larissa Kirschner. (2024). Short-Answer Grading for German: Addressing the Challenges. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-023-00383-w

q2 · i? · design · r2

Same-corpus evaluation experiments across the four German SAG corpora show a spread in weighted F-Score from 59.8 (ASAP-DE) to 72.1 (CREG) under identical architecture, supporting the claim that "prediction performance can vary quite widely across corpora" and cannot be reliably predicted from benchmarks.

"Second, we see clearly that for the same machine learning model architecture and underlying language model, prediction performance can vary quite widely across corpora."

Discussion¶

Related Claims¶

• Under unseen-answers evaluation, the German Transformer model's F-Scores are numerically similar to state-of-the-art Transformer results for English — related
• A fine-tuned SBERT Transformer SAG model for German approximates but does not beat hand-crafted feature-based models in unseen-question evaluation — related
• The predictive value of grade 3 screening benchmarks for MCAS ELA proficiency varies widely: 15 to 69 percent of students classified as on track did not reach MCAS proficiency, and students flagged at risk rarely met proficiency — related
• Science readability models trained on context-specific corpora fail to generalize to unseen test corpora in all three languages — a narrower finding that bears on this claim
• Supervised ML/DL Bloom question classifiers trained on one dataset drop substantially in weighted F1 when tested on unseen datasets — a broader claim this one bears on
• The German Transformer model's tokenization handles native-speaker morphology well but struggles with learner-language errors, possibly explaining lower CREG results — related