Validate automated video analytics and reach metric consensus before widespread adoption in surgical training¶

Strategy · All strategies
Evidence · 2 claims (2 for) · 1 study (1 review), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description¶

The review recommends that educators, curriculum developers and professional bodies support validation studies in training environments before widespread adoption of automated video analytics. It states that "Establishing robust validity of automated video analytics is essential if this innovation is to help shape future training" and that consensus must be reached on which operative performance metrics are most relevant for surgical trainees. Only after validation can evidence demonstrating impact on training be established.

Design Implications¶

Context¶

Requirements¶

• Robust validity evidence for analysis techniques and generated metrics; consensus on relevant performance metrics

Constraints¶

• Robust protocols and frameworks are needed to address patient concerns over data storage, access and potential breaches

Target Learners¶

• surgical trainees and fellows

Target Learning Goals¶

• operative performance assessment and feedback

Claims¶

• Automated video metrics correlate with validated expert-based assessment tools [+W]
• No included study applied automated video metrics longitudinally to training or assessed non-technical skills [+W]

Related Strategies¶

• Combine objective metrics with validated subjective tools and audio-visual recording to contextualize interpretation of surgical performance data
• Use cross-method validity to select the most useful and reliable assessment tools when constructing a surgical training curriculum.
• Prioritize longitudinal studies tracking trainees from simulation to real-world surgical performance

Examples¶

-

Key Sources¶

• Dick L, Boyle CP, Skipworth RJE, Smink DS, Tallentire VR, Yule S. (2024). Automated analysis of operative video in surgical training: scoping review. BJS Open. https://doi.org/10.1093/bjsopen/zrae124