In the first deployment, exam scores on the induction question were visibly lower than in previous semesters taught without the tool¶
Claim · All claims
Evidence · 1 study · designr3·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Fall 2025 exam scores on each part of the induction question were visibly lower than in the four prior semesters, and a significant proportion of students left the inductive step blank. → Matthew Keenan 2026
Evidence¶
Matthew Keenan 2026¶
Matthew Keenan, Nishant Kheterpal, Jean-Baptiste Jeannin, Cyrus Omar. (2026). Hazel Prover: A Classroom Proof Assistant for Learning Structural Induction. https://arxiv.org/abs/2608.23309 doi:10.48550/arxiv.2608.23309
q2 · i? · design · r3
Comparison of average midterm induction-question scores (Figure 3) across semesters before and during tool use. The authors note the exam question varied semester to semester, so it is impossible to truly know whether students performed worse or the question was harder; no effect size is printed.
"The scores were visibly lower in the Fall 2025 semester, the first semester of using the tool, than in previous semesters."
Discussion¶
Related Claims¶
- Requiring students to write out evaluation steps manually improved transfer to pen-and-paper exam performance between the two deployments — related
- CS1-CR exam scores showed a slight, statistically insignificant increase compared to prior semesters — reports the opposite
- Fall clusters of coding-question students trend toward higher exam scores than report-writer clusters — related
- After the design iteration, no student deleted an evaluation step, in contrast to frequent step deletion in the first deployment — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
first-deployment-exam-scores-lower |
title |
In the first deployment, exam scores on the induction question were visibly lower than in previous semesters taught without the tool |
description |
In the first deployment, exam scores on the induction question were visibly lower than in previous semesters taught without the tool |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — matthew-keenan-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.