Skip to content

Large-scale field study dataset and code release for AI-generated exam evaluation

Product or Programme · All products and programmes
Evidence · no claims cited

Description

A publicly released dataset and code package containing field-study student responses, AI-generated and comparison exam materials, and analysis code for evaluating course-tailored exams, produced by Calvin Isley and colleagues.

Components

  • Released dual-graded exam datasets, ablation grid, and grading, fine-tuning and analysis pipelines: The article releases two anonymised dual-graded exam datasets (CV exam, 570 students; ML exam, 1,038 students), the rubrics and reference solutions, one result workbook per configuration, and the analysis, grading and fine-tuning pipelines. The authors state that "Every number in the paper is regenerated from released artifacts by released code", making the released ledgers the authority for the manuscript. The artifact supports replication of the persona-collapse and fine-tuning findings. (Habibullah et al. (2026))
  • Two-instrument evaluation dataset: chemistry diagnostic and psychometric quantitative chapter with injected chatbot responses: The study's empirical base is a pair of authentic MCQ instruments from very different contexts. The first was "a chemistry diagnostic examination administered via a Moodle platform as a formative assessment activity" with 22 MCQs and 931 high-school respondents; the second was a 20-item quantitative chapter from a high-stakes psychometric entrance exam with over 4,800 respondents. Both are multi-modal, containing images/figures and formulas. GenAI responses were generated by running each of three premium chatbots 20 times per instrument as new sessions, with non-zero temperature, and non-answers counted as incorrect. (Alona Strugatski et al. (2025))

Claims

Key Sources

  • Calvin Isley, Joshua Gilbert, Evangelos Kassos, Michaela Kocher, Allen Nie, Emma Brunskill, Ben Domingue, Jake Hofman, Joscha Legewie, Teddy Svoronos, Charlotte Tuminelli, Sharad Goel. (2025). Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study. https://arxiv.org/abs/2508.08314
  • Habibullah, A., Alshoibi, Y., Alshiekh, M., Khan, S., & Khan, N. (2026). Where LLM graders succeed and break: Evidence from two computer-science exams. arXiv. https://arxiv.org/abs/2609.29333. doi:10.48550/arxiv.2609.29333
  • Alona Strugatski and Giora Alexandron. (2025). Applying Item Response Theory to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessments. LAK'25 pre-print. https://arxiv.org/abs/2412.02713
Page metadata
Field Value
type product
id large-scale-field-study-dataset-and-code-release-for-ai-generated-exam-evaluation
title Large-scale field study dataset and code release for AI-generated exam evaluation
description A publicly released dataset and code package containing field-study student responses, AI-generated and comparison exam materials, and analysis code for evaluating course-tailored exams, produced by Calvin Isley and colleagues.
status review
product_kind dataset
generated by process:settle-candidates on 2026-10-10
trust unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.