Inter-assessor agreement on OSCE scores is substantial, with a pooled intraclass correlation coefficient of 0.77¶

Claim · All claims
Evidence · 1 study · quant-synthesis r1 · q3
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q3 i? The pooled ICC between assessors was 0.77 (95% CI: 0.7, 0.83) with no statistical heterogeneity, indicating substantial interrater reliability. → Bobos Pavlos 2021

Evidence¶

Bobos Pavlos 2021¶

Bobos Pavlos, Pouliopoulou Dimitra V., Harriss Alexandra, Sadi Jackie, Rushton Alison, MacDermid Joy C. (2021). A systematic review and meta-analysis of measurement properties of objective structured clinical examinations used in physical therapy licensure and a structured review of licensure practices in countries with well-developed regulation systems. PLoS ONE. https://doi.org/10.1371/journal.pone.0255696

q3 · i? · quant-synthesis · r1

Random effects meta-analysis of intraclass correlation coefficients between OSCE assessors across included studies. The pooled estimate was "0.77 (95% CI: 0.7, 0.83)" with no heterogeneity, I2 = 0%. No standardised effect size was printed for this finding.

"The pooled estimate of Intra-class Correlation Coefficient (ICC) between assessors based on the available data was 0.77 (95% CI: 0.7, 0.83) with no statistical heterogeneity, I2 = 0% (Fig 6)."

Discussion¶

Related Claims¶

• Canada (excluding Quebec) is the only country of 17 requiring a clinical competency examination for licensure of domestic physical therapy graduates — related
• Standardised patient performance shows almost perfect inter-rater (kappa 0.84) and substantial intra-rater (kappa 0.75) agreement — related
• Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency — related
• An automated LLM assessor rates 3E clinical communication skills with agreement comparable to individual human raters — related