GenAI models' grades differ significantly across models, with GPTo1 grading more leniently than Llama2, Mistral7B and Cohere¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? The Friedman test showed statistically significant differences among models' scores, and Wilcoxon tests with Bonferroni correction found GPTo1 grades better than Llama2, Mistral7B and Cohere but comparable to PaLM2 and Claude3. → Pecuchova 2025 q2 i? Llama2, Mistral7B and Cohere showed no significant grade differences with near-zero RBC values, indicating practical equivalence at grade level. → Pecuchova 2025

Evidence¶

Pecuchova 2025¶

Pecuchova, J., Benko, Ľ., & Drlik, M. (2025). Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00517-2

q2 · i? · design · r2

Non-parametric comparisons in the evaluation phase (Friedman ANOVA N = 1885, df = 10, p < 0.01; Wilcoxon with Bonferroni correction); a negative RBC of -0.2791 suggested "GPTo1 grades with better marks than Cohere", while it "performs comparably to PaLM2 and Claude3".

"On the other hand, GPTo1 performs comparably to PaLM2 and Claude3 ( p > 0.05)."

Discussion¶

Related Claims¶

• Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades — related
• Statistical tests show significant grading differences among nearly all pairs of LLMs, with only 7 of 153 pairs statistically indistinguishable — possibly the same claim (merge candidate)
• The omnibus ANOVA on total QM score did not reach conventional significance, though a Friedman test suggested an overall difference with no significant pairwise Wilcoxon comparisons — related