ChatGPT could not execute R code for statistics MCQs, achieving 60% accuracy when its code was run with corrected dataset naming¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On MCQs requiring students to run R code, ChatGPT could not execute code itself and produced code with incorrect object names; when run with the dataset named correctly, accuracy was 60%. → Ivory 2026

Evidence¶

Ivory 2026¶

Ivory, M., Finnerty, S., Dunn, K., Philpot, R., & Warmelink, L. (2026). The Integrity of Psychology Assessments in the AI Age: A Critical Examination. Journal of Academic Ethics. https://doi.org/10.1007/s10805-026-09729-0

q2 · i? · design · r2

In statistics modules using R, ChatGPT lacked access to the datafile and could not execute code; the researchers ran its code after adding correct dataset naming, yielding 60% accuracy, which the authors note a student could easily improve by supplying column names.

"the accuracy was 60% for these types of questions, and it did not score higher mostly because it guessed at column names."

Discussion¶

Related Claims¶

• ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related
• Module 3 code execution success dropped to approximately 60% before a missing-dependency infrastructure fix, recovering to match the 86–87% rates of Modules 1 and 2 — related
• Poorly grounded GenAI self-study risks fabricated content and inferior learning relative to curated resources — related