AI-SSP backend architectures shifted from rule-based or NLP systems before 2022 to LLM-based systems in all studies published after 2022¶

Claim · All claims
Evidence · 1 study · review r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? All included studies published after 2022 relied partially or fully on LLMs for patient dialogue, scoring, or feedback generation, whereas earlier systems used rule-based NLP, scripted dialogue, decision trees, or keyword matching. → Zhang Wenjia (Stella) 2026

Evidence¶

Zhang Wenjia (Stella) 2026¶

Zhang Wenjia (Stella), Daniels Benjamin B, Mita Carol, Nguyen Hoang, Duong David B. (2026). AI for Clinical Competency Assessment: Scoping Review of Methods and Applications. JMIR Medical Education. https://doi.org/10.2196/92826

q3 · i? · review · r3

Review synthesis across the 20 unique systems charted by publication year and underlying AI model type, shown as a timeline figure. The review states that "After 2022, all included studies either partially or fully relied on LLMs" for dialogue, scoring, or feedback generation, while earlier systems used rule-based approaches such as approximately 2500 ChatScript rules.

"After 2022, all included studies either partially or fully relied on LLMs to power patient dialogue, scoring, or feedback generation."

Discussion¶

Related Claims¶

• CA prototype technologies shifted from rule-based and deterministic systems toward open-source tools, pre-trained LLMs, and retrieval-augmented generation pipelines — a broader claim this one bears on
• AWE technology has evolved through four distinct generations from traditional NLP-based systems to AI-driven LLM-integrated tools (2000–2024) — a broader claim this one bears on
• History-taking was the most frequently assessed competency in AI-SSP studies (52%), and assessed domains shifted over time toward interactional competencies — related
• AI-SSP clinical case development remained largely human-authored, with AI used mainly for interaction and scoring rather than case generation — related
• Twenty-one studies describing 20 unique AI-SSP systems for clinical competency assessment were published between 2008 and 2026, with 71% published in 2024 or later — related
• Clearest LLM-teaching signals appear in settings with clearly defined tasks, immediate feedback, and repeated practice, such as simulated patients and structured feedback — related
• The 11 included records spanned publication years 1991 to 2022, with most (n = 6) published in 2016 or later. — related
• Across the 25-year span of reviewed studies, educational approaches shifted from content-based to competency-based learning — related