Claims

Estimating each grader's bias alone accounts for 95% of the RMSE reduction, while modeling grader reliability contributes little with only four grades per student

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2013 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2016: tool and model accuracy changes quickly

Subclaims

Reading the evidence codes

q is the design tier, i the impact, n the sample, and the kind with its r the rigour judged within that kind: what each code means.

This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.

q2 i? The single change of estimating each grader's bias (Model PG1-bias) is responsible for 95% of the reduction in RMSE; modeling grader precision does little to improve performance. → Chris Piech 2013

Evidence

Chris Piech 2013

Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, Daphne Koller. (2013). Tuned Models of Peer Assessment in MOOCs. arXiv. https://arxiv.org/abs/1307.2579

q2 · i? · design · r2

Simulation-based evaluation on the HCI datasets (Table 2). The article reports that the bias-only model "is responsible for 95% of our reduction in RMSE", and that on a synthetic dataset generated from Model PG1, reliability is hard to estimate with only four grades per student, becoming notable only with more than 10 assignments graded.

"But the single change that provides the most significant gains in accuracy is obtained by estimating each grader's bias (Model PG1-bias ). This simple model is responsible for 95% of our reduction in RMSE."

Discussion

Page metadata
Field Value
type claim
id grader-bias-estimation-dominates-accuracy-gains
title Estimating each grader's bias alone accounts for 95% of the RMSE reduction, while modeling grader reliability contributes little with only four grades per student
description Estimating each grader's bias alone accounts for 95% of the RMSE reduction, while modeling grader reliability contributes little with only four grades per student
status review
evidence_strength moderate
newest_evidence 2013
recheck_by 2016
generated by process:wiki-ingest on 2026-10-16
trust unverified — no one has confirmed this page's content
sources 1 — chris-piech-2013

Frontmatter is the page's machine-readable record — see the Schema & Guide.