Checklists show reasonable validity and reliability but have an early ceiling effect and are most appropriate for novice learners in formative settings¶
Claim · All claims
Evidence · 1 study · reviewr3·q2
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2023: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Checklists demonstrate reasonable construct validity, concurrent validity, and reliability, but do not capture quality of performance and show an early ceiling effect, making them most appropriate for novice learners in formative settings. → James H.K. 2020
Evidence¶
James H.K. 2020¶
James H.K., Chapman A.W., Pattison G.T.R., Fisher J.D., Griffin D.R. (2020). Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review. JBJS Reviews. https://doi.org/10.2106/JBJS.RVW.19.00167
q2 · i? · review · r3
The review's narrative synthesis of checklist studies (20 studies used task-specific checklists). It reports reasonable construct, concurrent validity and reliability, while noting rigid binary scoring, inability to capture performance quality, and an early ceiling effect.
"They show reasonable construct validity68,77,96,98, concurrent validity37,77,96,102,103, and reliability37,88,114. With their limitations in mind, checklists are perhaps most appropriate for novice learners in a formative setting1."
Discussion¶
Related Claims¶
- The 105 included studies of orthopaedic technical-skills assessment were predominantly conducted in simulated settings with 2,088 resident participants
- Global rating scales discriminate between performance levels without a ceiling effect, making them useful for high-stakes summative assessment of advanced residents — related
- Most identified assessment tools demonstrate construct or concurrent validity, but predictive validity is absent across tool categories — related
- Global rating scales by expert practitioners are as reliable as item-specific checklists and less labour intensive to devise — related
- No identified procedural skills assessment tool has established construct validity; only face and content validity are reported — reports the opposite
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-11-01 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.