Modified Angoff with reality check achieves excellent inter-rater reliability¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The Modified Angoff with reality check achieved excellent reliability (r = 0.917) while the Angoff (Yes/No) yielded the weakest and most variable results (r = 0.536). → Kannan 2025

Evidence¶

Kannan 2025¶

Kannan, S., & Gowri, S. (2025). Angoff methods in standard setting in health professional education: a systematic review and meta-analysis. BMC Medical Education, 25, 1727. doi:10.1186/s12909-025-08300-6

q2 · i? · quant-synthesis · r2

The meta-analysis pooled inter-rater reliability coefficients (Fisher's z-transformed) across Angoff variants. The Modified Angoff with reality check achieved "excellent reliability (r = 0.917)" while the Angoff (Yes/No) yielded "the weakest and most variable results (r = 0.536)".

"The Modified Angoff with reality check achieved excellent reliability (r = 0.917) while the Angoff (Yes/No) yielded the weakest and most variable results (r = 0.536), a finding confirmed by bootstrap analyses."

Discussion¶

Related Claims¶

• Angoff (Yes/No) variant shows the highest variability in cut scores — related
• Angoff and Modified Angoff standard-setting show good inter-rater reliability in dental education, with reported coefficients ranging from 0.65 to 0.94 — a broader claim this one bears on
• Percentage rating format in the Modified Angoff method yields higher reliability than the Yes/No format, with ICCs of 0.48–0.65 versus a maximum of 0.41 — related