Multi-label classification of open-ended course feedback was challenging even for expert human annotators who developed the label set, with average inter-rater Jaccard similarity of 81.24%¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Four expert human annotators averaged 81.24% Jaccard similarity when multi-label tagging 2,500 survey comments, which the authors interpret as evidence the task was challenging even for experts. → Michael J. Parker 2024 q3 i? Full four-rater agreement was reached on a minority of rows: 1,413 (57%) of 2,500 rows had all 4 human raters in agreement across all selected labels. → Michael J. Parker 2024 (2)

Evidence¶

Michael J. Parker 2024¶

Michael J. Parker, Caitlin Anderson, Claire Stone, YeaRim Oh. (2024). A Large Language Model Approach to Educational Survey Feedback Analysis. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-024-00414-0

q3 · i? · design · r2

Inter-rater agreement analysis of 2,500 end-of-course survey comments from online biomedical science courses, labeled by the four expert authors, found average Jaccard similarity of "81.24% (Table 4)" across rater pairs. The article states this suggests "a challenging task even for expert human annotators".

"The average Jaccard similarity coefficient including all 2,500 rows (averaged across the six unique pairings of the four human raters for all rows) was 81.24% (Table 4)"

Michael J. Parker 2024 (2)¶

Michael J. Parker, Caitlin Anderson, Claire Stone, YeaRim Oh. (2024). A Large Language Model Approach to Educational Survey Feedback Analysis. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-024-00414-0

q3 · i? · design · r2

The same inter-rater agreement analysis reports that "1,413 (57%) of 2,500 rows had all 4 human raters in agreement" and 1,572 (63%) had majority agreement, quantifying the inherent difficulty of the multi-label tagging task.

"1,413 (57%) of 2,500 rows had all 4 human raters in agreement across all selected labels and 1,572 (63%) had majority (3 of 4) agreement on all selected labels."

Discussion¶

Related Claims¶

• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related