Prioritize predictive-validity research, benchmarking of existing tools, and integration of kinematic data and AI in surgical skill assessment¶

Strategy · All strategies
Evidence · 6 claims (6 for) · 1 study (1 review), q3 · 0 of 1 report an effect size · 6 claims rest on one study

Description¶

The review's forward-looking recommendation is that future work should fill the predictive-validity gap, benchmark already available tools for national curricula, and integrate advanced technologies. The authors write that "Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment" within gynecological advanced training programs. They point to examples in other specialties, such as correlating kinematic data from recording devices with technical performance, and to surgical data science approaches including deep neural networks, while flagging the absence of large-scale datasets as a key challenge.

Design Implications¶

Context¶

Requirements¶

• Robust training of AI models requires large-scale datasets representing domains of variation such as experience level, subtask and instrument type

Constraints¶

• Currently only limited clinical datasets are available; existing pass/fail scores have not been implemented for credentialing in training curricula

Target Learners¶

• surgical trainees in minimally invasive gynecological surgery
• gynecological surgical educators and training-program designers

Target Learning Goals¶

• standardized, objective assessment of surgical technical skills
• proficiency-based progression and credentialing in surgical curricula

Claims¶

• A systematic review of minimally invasive gynecological surgery identified 37 objective assessment tools across 72 included studies [+W]
• The majority of automated performance metrics in gynecological surgery have limited validity evidence, hindering transferability outside research settings [+W]
• Most identified assessment tools demonstrate construct or concurrent validity, but predictive validity is absent across tool categories [+W]
• No objective assessment tool in minimally invasive gynecological surgery showed predictive validity correlating scores with patient outcomes [+W]
• Only two of 37 objective assessment tools in minimally invasive gynecological surgery showed substantial evidence of validity [+W]
• Procedure-specific assessment tools hold greater educational potential than readily available global tools, though few provide benchmark scores [+W]

Related Strategies¶

• Combine objective metrics with validated subjective tools and audio-visual recording to contextualize interpretation of surgical performance data
• Integrate validated data-driven assessment tools early in surgical training and maintain use throughout residency with pre-course benchmarks and immediate feedback

Examples¶

-

Key Sources¶

• Tesfai FM, Nagi J, Morrison I, Boal M, Olaitan A, Chandrasekaran D, Stoyanov D, Lanceley A, Francis N. (2024). Objective assessment tools in laparoscopic or robotic‐assisted gynecological surgery: A systematic review. Acta Obstet Gynecol Scand. https://doi.org/10.1111/aogs.14840