gpt-4o graded student submissions most correctly in almost all assignments, but gemini-1.0-pro graded the boolean-algebra assignment most correctly¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across the four assignments, gpt-4o showed the highest grader correctness in almost all assignments except A3. → Xinming Yang 2025 q2 i? For assignment A3 on boolean algebra transformations, gemini-1.0-pro achieved much higher correctness than both gpt-3.5-turbo and gpt-4o. → Xinming Yang 2025 (2)

Evidence¶

Xinming Yang 2025¶

Xinming Yang, Haasil Pujara, Jun Li. (2025). Learning by Teaching: Engaging Students as Instructors of Large Language Models in Computer Science Education. Published as a conference paper at COLM 2025. https://arxiv.org/abs/2508.05979 doi:10.48550/arxiv.2508.05979

q2 · i? · design · r2

Log-based observational analysis of grader correctness on the four assignments' submissions, comparing grader results with manual grading and combining true positives and true negatives. The article reports "In almost all assignments (except A3), gpt-4o demonstrates the highest correctness." No effect size is printed.

"In almost all assignments (except A3), gpt-4o demonstrates the highest correctness."

Xinming Yang 2025 (2)¶

Xinming Yang, Haasil Pujara, Jun Li. (2025). Learning by Teaching: Engaging Students as Instructors of Large Language Models in Computer Science Education. Published as a conference paper at COLM 2025. https://arxiv.org/abs/2508.05979

q2 · i? · causal · r1

Same grader-correctness log analysis, assignment A3. The article calls this "a tentative observation specific to our test cases" suggesting a potential strength of gemini-1.0-pro on this type of logical reasoning problem. No effect size is printed.

"for assignment A3, which focused on applying transformations in boolean algebra, gemini-1.0-pro achieved much higher correctness than both gpt-3.5-turbo and gpt-4o."

Discussion¶

Related Claims¶

• Rubric-enhanced prompting (Variant 2) raises LLM assigned scores relative to the no-rubric baseline, with model-specific gaps at particular taxonomy levels — related
• Gemini and Gemini Pro show the highest internal grading consistency across file upload volumes, with no clear monotonic lost-in-the-middle effect — related
• Running the Socrates playground and grader incurred low cost, with even gpt-4o consuming only $169.90 — related
• LLMs differ systematically in grading strictness, forming liberal, restrictive, and balanced grading styles on programming assignments — related
• LLM grading patterns cluster into six groups aligned with vendor families, with highest correlations within same-vendor model families — related