Rubrics with parallel sentence structure, unambiguous wording and well-specified dimensions have lower grading variance, and revising rubrics decreased median grading error from 12.4% to 9.9%¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r? · q2
Currency · newest study 2015 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Rubrics using parallel sentence structure, unambiguous wording and well-specified dimensions showed lower variance in peer grading. → Kulkarni 2015 q2 i? After rubric revision, median grading error decreased from 12.4% to 9.9%. → Kulkarni 2015 (2)

Evidence¶

Kulkarni 2015¶

Kulkarni, C., Koh Pang Wei, Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., Klemmer, S. R. (2015). Peer and Self Assessment in Massive Online Classes. In: Design Thinking Research, Understanding Innovation. Springer, Cham. https://doi.org/10.1007/978-3-319-06823-7_9

q2 · i? · associational · r?

Analysis of grading variance across rubric items in the online class's peer assessment data. The paper reports that rubrics with "a parallel sentence structure, unambiguous wording and well-specified dimensions have lower variance"; no effect size is printed.

"We find that rubrics that use a parallel sentence structure, unambiguous wording and well-specified dimensions have lower variance."

Kulkarni 2015 (2)¶

Kulkarni, C., Koh Pang Wei, Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., Klemmer, S. R. (2015). Peer and Self Assessment in Massive Online Classes. In: Design Thinking Research, Understanding Innovation. Springer, Cham. https://doi.org/10.1007/978-3-319-06823-7_9

q2 · i? · causal · r1

Outcome of the rubric-revision intervention in the online class. The paper reports that "median grading error decreased from 12.4 to 9.9 %" after revision; no standardized effect size is printed.

"After revising rubrics, median grading error decreased from 12.4 to 9.9 %."

Studies bearing on this claim¶

Do raters applying a rubric agree, and what in the rubric's design brings their scores closer? This table gathers the studies behind this claim and the 5 narrower claims that bear on it: 5 distinct studies (3 design, 1 associational, 1 qualitative), q2, 0 of 5 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Raters differed significantly in severity/leniency in the MFRM analysis — the general claim this finding bears on, with the studies behind it
• K-12 students across five schools valued CyberScholar's detailed, specific, rubric-aligned feedback as helpful for identifying areas to revise — related
• Two rubric dimensions (claims and evidence, comparison) showed low inter-scorer consistency in the pilot — related
• Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable — related
• The VCP rubric was difficult for teams to use because it lacked standardized calibration and its process and content categories were unclear — related