Bard unexpectedly cited references including the 2021 AAMC SJT practice exam booklet, suggesting potential use as a reference manager despite its inaccuracy¶

Claim · All claims
Evidence · 1 study · design r2 · q1
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q1 i? Bard, despite scoring worst, responded to queries with references justifying its responses, among which it cited the 2021 AAMC SJT practice exam booklet. → Cadiente 2024

Evidence¶

Cadiente 2024¶

Cadiente, A., Chen, J., Kasselman, L., & Pilkington, B. (2024). Large Language Models take on the AAMC Situational Judgment Test: Evaluating Dilemma-Based Scenarios. Research Square preprint. https://doi.org/10.21203/rs.3.rs-4560463/v1

q1 · i? · design · r2

Authors' qualitative observation (type e) from their work with Bard during the study; the frequency of citation of the question set was not recorded and was outside the study's scope. The article recommends Bard be used with caution in assessing social situations but notes possible usage as a reference manager.

"Despite its comparative inaccuracy, it unexpectedly responded to queries with references justifying its response. Among its references, the 2021 AAMC SJT practice exam booklet was cited."

Discussion¶

Related Claims¶

• ChatGPT-4.0 outperformed ChatGPT-3.5 and Bard on the AAMC SJT in both raw and adjusted accuracy — related
• ChatGPT fabricates references: the authors caught it inventing a non-existent 'Ribeiro and Vala, 2020' article with full bibliographic details — related