GPT models achieved high polarity agreement with instructors only for positive feedback on the Topic aspect, and could not reliably assess the other rubric aspects¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? The highest agreement was Topic positive polarity (precision 0.85, recall 0.98, F1 0.91 for GPT-4); other aspects showed low precision, recall and F1. → Dai 2024

Evidence¶

Dai 2024¶

Wei Dai, Yi-Shan Tsai, Jionghao Lin, Ahmad Aldino, Hua Jin, Tongguang Li, Dragan Gašević, Guanliang Chen. (2024). Assessing the Proficiency of Large Language Models in Automatic Feedback Generation: An Evaluation Study. Computers and Education: Artificial Intelligence, 7, 100299. Preprint submitted to Elsevier. https://doi.org/10.1016/j.caeai.2024.100299

q2 · i? · design · r2

Polarity agreement analysis with instructor feedback as ground truth across five rubric aspects (Table 8). "the GPT models achieved the highest agreement score with human instructors (precision: 0.85, recall: 0.98, F1: 0.91) in positive polarity for the aspect of Topic".

"the GPT models achieved the highest agreement score with human instructors (precision: 0.85, recall: 0.98, F1: 0.91) in positive polarity for the aspect of Topic"

Discussion¶

Related Claims¶

• Feedback generated by GPT-3.5 and GPT-4 was significantly more readable than instructor-generated feedback on open-ended writing tasks — related
• GPT-4 demonstrated higher reliability of feedback than GPT-3.5, with higher F1 agreement with instructors on most assessment aspects — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — related
• A human assessor agreed with the majority of GPT-generated feedback comments on assessment aspects that instructors had not commented on — related
• NLP-based feedback generation for textual student answers in large courses reduced grading effort by up to 85% with high precision (reported from Bernius et al.) — reports the opposite
• In a self-referential framework, LLMs, particularly GPT-4.1, tend to outperform a single operational human rater at identifying relative weaknesses (negative feedback) across several Language and Content aspects — related
• GPT-4 achieved almost perfect inter-rater reliability with faculty after three iterations of rubric engineering on three of four medical school OEQ questions — related