Messick's five-source validity framework for surgical assessment tools¶

Research Method · All research methods
Evidence · 6 claims (5 for, 1 mixed) · 4 studies (2 quant-synthesis, 2 review), q2–q3 · 1 of 4 report an effect size · 6 claims rest on one study

Description¶

The review applies a scoring tool based on the five sources of validity originally described by Messick: content, response process, internal structure, relation to other variables, and consequences. Assessment tools should ideally be supported by validity evidence from each of the five sources, though not necessarily within the same paper. The review used Ghaderi's toolbox, with scores from 0 to 15 points, to assess the validity evidence reported for each of the 31 identified assessment tools.

Accounts¶

• Messick's five-source validity framework for surgical assessment tools: The review applies a scoring tool based on the five sources of validity originally described by Messick: content, response process, internal structure, relation to other variables, and consequences. Assessment tools should ideally be supported by validity evidence from each of the five sources, though not necessarily within the same paper. The review used Ghaderi's toolbox, with scores from 0 to 15 points, to assess the validity evidence reported for each of the 31 identified assessment tools. (Hertz P et al. (2024))
• Contemporary five-aspect validity framework (adapted from Messick) for appraising surgical assessment tools: The review applies a contemporary framework of validity, adapted from Messick's validity framework, with five aspects: content, response process, internal structure, relationship to other variables, and consequences. Each aspect is scored 0–3, giving a maximum of 15, banded as limited (1–5), moderate (6–10) or substantial (11–15) validity. The scoring system was adapted from Beckman, Ghaderi, Haug and Grüter and modified for this review; separate scoring definitions are provided for manual tools and APMs, e.g. for APMs "Performance metrics were recorded by the simulator (i.e., eliminating rater bias)". The framework structures the review's per-tool validity tables. (Tesfai FM et al. (2024))
• Five-domain validity evidence framework for appraising surgical assessment tools: The review applies a five-domain validity framework, based on Messick's unitary view of validity and operationalized by Ghaderi et al, in which each assessment tool is scored on content, response process, internal structure, relation to other variables, and consequences, with a maximum score of 15. Two independent reviewers scored each of the 24 tools on these domains, with disagreements resolved by consensus. The framework is presented as having "good interrater reliability" when used in prior surgical specialties' reviews. (Li Yibo et al. (2024))
• Messick's unified construct validity framework for evaluating robotic VR simulators as assessment and selection tools: The article applies Messick's framework, which "conceptualizes all validity under one overarching framework of construct validity", to robotic VR simulators. Within it, concurrent validity asks how accurately simulator performances predict current intraoperative performance, informing credentialing, while predictive validity asks how accurately they predict future performance, informing selection of future robotic surgeons. The review uses this framework to organize its predictability studies and to argue that terminology around validity is often misunderstood and misused. (Schmidt MW et al. (2021))
• Messick's modern validity framework with Beckman ratings and McGaghie-style translational outcomes (LoE 1-5) for evaluating surgical simulators: The review applies a two-part evaluation framework to vascular surgery simulators. Validity evidence is judged against Messick's five sources — content, response process, internal structure, relations to other variables, and consequences — each rated 0-2 with Beckman's scale, where 2 means "Data strongly supports the source of validity". Effectiveness is scored with an adapted McGaghie translational outcomes hierarchy from LoE 1 (trainee satisfaction) to LoE 5 (collateral effects such as "Cost saving; skill retention"). The authors argue modern unitary validity definitions are "vital in the in-depth assessment of simulators" and recommend the framework as standard practice. (Haiser Alexander et al. (2022))

Claims¶

• Only two of 37 objective assessment tools in minimally invasive gynecological surgery showed substantial evidence of validity [+M]
• Validity evidence for procedure-specific orthopaedic surgery assessment tools is low, with total scores ranging from 1 to 9 of a maximum of 15 [+M]
• Across the five validity domains, tools scored highest on content and very poorly on consequences, with 15 of 24 tools scoring 0 in the consequences domain [+M]
• Robotic VR simulator performance is positively correlated with intraoperative performance, with a pooled correlation of r = 0.67 (95% CI 0.22 to 0.88) [+M]
• Correlations between simulator task performance and intraoperative GEARS domains vary by task, and one study found no correlation between dVSS scores and intraoperative GEARS scores [~M]
• Only seven studies of vascular surgery simulation achieved a level of effectiveness greater than 3/5, with the majority scoring LoE 1-2/5 [+M]

Related Research Methods¶

-

Key Sources¶

• Hertz P , Rattenborg S , Haug TR , Houlind K , Konge L , Bjerrum F . (2024). Training and assessment for colorectal surgery and appendicectomy– a systematic review. Colorectal Dis. https://doi.org/10.1111/codi.16905
• Tesfai FM, Nagi J, Morrison I, Boal M, Olaitan A, Chandrasekaran D, Stoyanov D, Lanceley A, Francis N. (2024). Objective assessment tools in laparoscopic or robotic‐assisted gynecological surgery: A systematic review. Acta Obstet Gynecol Scand. https://doi.org/10.1111/aogs.14840
• Li Yibo, Chan Robert, Menon Matthew R.G., Ryan Joanna F., Mador Brett, Campbell Sandra M., Turner Simon R. (2024). Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review. JAAOS Global Research & Reviews. https://doi.org/10.5435/JAAOSGlobal-D-23-00065
• Schmidt MW, Köppinger KF, Fan C, Kowalewski K-F, Schmidt LP, Vey J, Proctor T, Probst P, Bintintan VV, Müller-Stich B-P, Nickel F. (2021). Virtual reality simulation in robot-assisted surgery: meta-analysis of skill transfer and predictability of skill. BJS Open. https://doi.org/10.1093/bjsopen/zraa066
• Haiser Alexander, Aydin Abdullatif, Kunduzi Basir, Ahmed Kamran, Dasgupta Prokar. (2022). A Systematic Review of Simulation-Based Training in Vascular Surgery. The Journal of Surgical Research. https://doi.org/10.1016/j.jss.2022.05.009