AI-based assessment systems show strong internal performance and early real-time deployment for critical view of safety detection, but lack external validation and clinical impact evidence¶

Claim · All claims
Evidence · 1 study (3 entries) · review r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Internal structure evidence was reported in only 5 of 13 AI studies (38.5%), and generalisability was limited across the majority (8/13 AI studies, 61.5%). → Mainwaring Elizabeth M. A. 2026 q3 i? No study demonstrated consequential validity for AI-based assessment; there was no evaluation of impact on surgical performance, training progression, or patient safety outcomes such as bile duct injury. → Mainwaring Elizabeth M. A. 2026 (2) q3 i? AI-based CVS detection systems are the only LC assessment tools to demonstrate early real-time intraoperative deployment, in 3 systems. → Mainwaring Elizabeth M. A. 2026 (3)

Evidence¶

Mainwaring Elizabeth M. A. 2026¶

Mainwaring Elizabeth M. A., Guidozzi Nadia, James Jonnie P., Pantling Jai, Menon Nainika, Salem Riadh, Markar Sheraz R. (2026). Objective technical skills assessment in laparoscopic cholecystectomy: a systematic review of manual, kinematic, and AI-based tools. Surgical Endoscopy. https://doi.org/10.1007/s00464-026-13309-z

q3 · i? · review · r3

The AI-tools synthesis reports limited internal-structure evidence across the 13 AI studies: "Internal structure evidence was reported in only 5 of 13 AI studies (38.5%)", with direct skill classification models presenting the most consistent metrics. Generalisability was limited in 8/13 (61.5%) of studies, mostly trained on small single-centre datasets.

"Internal structure evidence was reported in only 5 of 13 AI studies (38.5%), direct skill classification models presenting the most consistent evidence, where performance metrics such as accuracy, area under the curve, and cross-validation results were commonly presented."

Mainwaring Elizabeth M. A. 2026 (2)¶

Mainwaring Elizabeth M. A., Guidozzi Nadia, James Jonnie P., Pantling Jai, Menon Nainika, Salem Riadh, Markar Sheraz R. (2026). Objective technical skills assessment in laparoscopic cholecystectomy: a systematic review of manual, kinematic, and AI-based tools. Surgical Endoscopy. https://doi.org/10.1007/s00464-026-13309-z

q3 · i? · review · r3

Across Messick's framework the review found that "There was no evaluation of the impact of AI-based assessment on surgical performance, training progression, behavioural modification, or patient safety outcomes"; reported benefits were limited to theoretical or anticipated improvements in feedback, standardisation, and decision support.

"There was no evaluation of the impact of AI-based assessment on surgical performance, training progression, behavioural modification, or patient safety outcomes such as bile duct injury."

Mainwaring Elizabeth M. A. 2026 (3)¶

Mainwaring Elizabeth M. A., Guidozzi Nadia, James Jonnie P., Pantling Jai, Menon Nainika, Salem Riadh, Markar Sheraz R. (2026). Objective technical skills assessment in laparoscopic cholecystectomy: a systematic review of manual, kinematic, and AI-based tools. Surgical Endoscopy. https://doi.org/10.1007/s00464-026-13309-z

q3 · i? · review · r3

The discussion notes that "AI-based CVS detection systems are the only tools to demonstrate early real-time intraoperative deployment", across the three AI systems deployed intraoperatively in real time, of which two had prospective validation cohorts per the results section.

"Notably, AI-based CVS detection systems are the only tools to demonstrate early real-time intraoperative deployment [41, 48, 51]."

Discussion¶

Related Claims¶

• Existing GenAI assessment frameworks (traffic light, two-lane, AIAS) remain broad, lack discipline-specific applicability, and are unvalidated in authentic settings — a broader claim this one bears on
• Perfect AI execution cannot remove four limits: accuracy does not settle end-setting, safety does not confer legitimacy, detection does not secure integrity, and outputs do not replace participation — related
• Kinematic assessment systems for laparoscopic cholecystectomy are weakly validated, restricted to small datasets, and capture only a narrow component of operative competence — related
• Manual assessment tools have the largest body of validity evidence in laparoscopic cholecystectomy but show variable reliability and no links to patient outcomes — related
• No objective assessment tool for laparoscopic cholecystectomy demonstrates comprehensive validity across all Messick domains, so no gold standard exists — a broader claim this one bears on
• Regulatory oversight by bodies such as the US FDA is presented as critical for fostering trust in AI among clinicians and patients — related
• No included study applied automated video metrics longitudinally to training or assessed non-technical skills — related
• Educational utility was not explicitly assessed for most tools, and no studies reported data on educational impact — related