Situational judgement tests show a statistically significant but negligible-magnitude effect on early program academic results (d=0.170), while personality test evidence was insufficient for meta-analysis¶

Claim · All claims
Evidence · 1 study (2 entries) · quant-synthesis r2 · q2–q3 · i0 negligible
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i0 SJT had a statistically significant effect size of 0.170 on early program academic results, a magnitude below d = 0.2. → Huang Pin-Hsiang 2024 q2 i? Extracted personality test effect sizes were insufficient to conduct a meta-analysis for any outcome. → Huang Pin-Hsiang 2024 (2)

Evidence¶

Huang Pin-Hsiang 2024¶

Huang Pin-Hsiang, Arianpoor Arash, Taylor Silas, Gonzales Jenzel, Shulruf Boaz. (2024). Insights into undergraduate medical student selection tools: a systematic review and meta-analysis. Journal of Educational Evaluation for Health Professions. https://doi.org/10.3352/jeehp.2024.21.22

q3 · i0 · quant-synthesis · r2

Random-effects meta-analysis of SJT studies (3 articles, 10 effect sizes), with printed pooled value d = 0.170, which is below d = 0.2 and therefore coded as negligible magnitude despite statistical significance. The discussion notes scarcity: "only 7 effect sizes from a single study and just one from another".

"While SJT had an effect size of less than 0.2 on early program academic results, its effect reached statistical significance (0.170; 95% CI, 0.032 to 0.308)."

Huang Pin-Hsiang 2024 (2)¶

Huang Pin-Hsiang, Arianpoor Arash, Taylor Silas, Gonzales Jenzel, Shulruf Boaz. (2024). Insights into undergraduate medical student selection tools: a systematic review and meta-analysis. Journal of Educational Evaluation for Health Professions. https://doi.org/10.3352/jeehp.2024.21.22

q2 · i? · quant-synthesis · r2

Data extraction across included studies found only one effect size for personality tests (PQA, MMPI cells in Table 2), precluding pooled estimation. No effect size conclusion is drawn.

"For personality tests, the extracted effect sizes were insufficient to conduct a meta-analysis for any of the outcomes of interest."

Discussion¶

Related Claims¶

• ChatGPT-4.0 outperformed ChatGPT-3.5 and Bard on the AAMC SJT in both raw and adjusted accuracy — related
• Written knowledge tests, OSCEs and situational judgement tests all show predictive validity for later clinical performance — a broader claim this one bears on
• Multiple mini-interviews significantly predict early clinical exams (d=0.417) and early academic results (d=0.195), while panel interviews show no significant predictive effects — related
• The three LLMs showed fair inter-model agreement on the AAMC SJT, agreeing most strongly on incorrect answers — related
• Within aptitude test subtests, verbal reasoning (d=0.704) and quantitative reasoning (d=0.643) best predict early program academic results, while abstract and verbal reasoning predict end-of-program outcomes — related