Five judges classifying 63 test units reached average inter-judge correlation of 0.542 ± 0.019 on the two scales¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 1923 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The uniformity of judgment among the five judges, indicated by average judge-with-judge correlation, is 0.542 ± 0.019. → Herring 1923

Evidence¶

Herring 1923¶

Herring, J. P. (1923). Verbal and Abstract Elements in Intelligence Examinations. The Journal of Educational Psychology. https://psycnet.apa.org

q2 · i? · associational · r2

Classification study in which five judges assigned each of 63 page-length test units a position from 1 to 7 on both scales; "The uniformity of judgment is indicated by the average correlation, judge with judge, of 0.542 ± 0.019."

"The uniformity of judgment is indicated by the average correlation, judge with judge, of 0.542 ± 0.019."

Discussion¶

Related Claims¶

• A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
• Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement — related
• LLM-as-judge automated evaluation can achieve human-level agreement when carefully validated — related
• The test's exercises were validated by six expert judges who reached practical agreement after modification — related
• Judges' ratings on the Thorndike handwriting scale agree closely, and averaged ratings give a constant determination — related