Automated assessment marks show very strong positive correlation with expert tutor marks on search algorithm exercises¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q3
Currency · newest study 2016 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? There was a very strong, positive correlation between tutor's marks and automated marking system ones (r=.96, n=400, p<.0005). → Grivokostopoulou 2016 q3 i? Linear regression showed r²=.934, meaning 93.4% of the total variation in tutor marks can be explained by the linear relationship with automated marker scores. → Grivokostopoulou 2016 (2)

Evidence¶

Grivokostopoulou 2016¶

Grivokostopoulou, F., Perikos, I., & Hatzilygeroudis, I. (2016). An Educational System for Learning Search Algorithms and Automatically Assessing Student Performance. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-016-0116-x

q3 · i? · design · r2

Corpus of 400 student answers to search algorithm exercises from 80 undergraduate students, jointly marked by three experienced tutors on a 100-point scale as gold standard, then assessed by the automated marker. Pearson correlation calculated between the two sets of marks. The study found "a very strong, positive correlation between tutor's marks and automated marking system ones."

"The results showed that there was a very strong, positive correlation between tutor's marks and automated marking system ones, which was statistically significant (r =. 9 6 , n = 400, p < .0005)."

Grivokostopoulou 2016 (2)¶

Grivokostopoulou, F., Perikos, I., & Hatzilygeroudis, I. (2016). An Educational System for Learning Search Algorithms and Automatically Assessing Student Performance. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-016-0116-x

q3 · i? · associational · r2

Linear regression on the same 400-answer corpus with automated marker scores as independent variable and tutor marks as dependent variable. The regression yielded y = 9.697 + 0.894x with r²=.934, indicating "93.4 % of the total variation in y can be explained by the linear relationship between x and y."

"A simple linear regression application to both corpuses of student answers resulted in the following equation: y = 9.697 + 0.894x with r2 = . 934."

Discussion¶

Related Claims¶

• Peer marks correlate with teacher marks, with a mean correlation of 0.69 across studies in a meta-analysis — a broader claim this one bears on
• Vocabulary test scores correlate moderately with teachers' final Latin grades, averaging .56 for Latin-English and .66 for English-Latin — a broader claim this one bears on
• Rank-based metrics can be misleading for analytic assessment: systems prompted for holistic scoring achieve strong, sometimes stronger, correlations with analytic scores than when prompted for those aspects — related
• Holistic peer feedback correlated significantly with tutor marks while specific-criteria rubric feedback did not in the PEARS system — related
• An accountability feature allowing students to grade received reviews improved the correlation between expert and peer marks in peerScholar — related