The Procedure Based Assessment shows poor engagement and deficient orthopaedic utility evidence, and the authors propose it is no longer appropriate for summative assessment¶

Claim · All claims
Evidence · 1 study · review r3 · q2
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2023: tool and model accuracy changes quickly

Subclaims¶

q2 i? A national survey found poor resident engagement with PBAs, with more than half agreeing that completing PBAs is nothing but a form-filling exercise, and the authors propose the PBA is no longer appropriate for summative assessment. → James H.K. 2020

Evidence¶

James H.K. 2020¶

James H.K., Chapman A.W., Pattison G.T.R., Fisher J.D., Griffin D.R. (2020). Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review. JBJS Reviews. https://doi.org/10.2106/JBJS.RVW.19.00167

q2 · i? · review · r3

The review reports a national survey of U.K. trauma and orthopaedic resident attitudes toward PBAs, citing Hunter et al. The review attributes the poor engagement to the tool being burdensome, having a coarse binary rating scale, and causing assessment fatigue.

"A national survey of trauma and orthopaedic resident attitudes toward procedure-based assessments (PBAs) in the U.K. found that more than half agreed or strongly agreed with the statement “completing PBAs is nothing but a form-filling exercise,” 60% agreed or strongly agreed that there are “barriers to the successful use of PBAs by residents”120, and only one-third believed that they should be used for high-stakes assessment in training"

Discussion¶

Related Claims¶

• Only 24 procedure-specific assessment tools exist across orthopaedic surgery, covering only trauma, sports medicine, spine, and upper extremity subspecialties
• Global rating scales discriminate between performance levels without a ceiling effect, making them useful for high-stakes summative assessment of advanced residents
• No current technical skills-assessment tool in orthopaedic surgical training fulfills accepted criteria for effective assessment — related
• Performance-based assessments score low on generalizability, reliability, and standardization, with rater effects influencing scores — related