Requiring students to write out evaluation steps manually improved transfer to pen-and-paper exam performance between the two deployments¶
Claim · All claims
Evidence · 1 study · designr3·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? The proportion of students leaving the exam induction question blank was significantly higher in the first deployment than the second, suggesting better pen-and-paper confidence after the design iteration. → Matthew Keenan 2026
Evidence¶
Matthew Keenan 2026¶
Matthew Keenan, Nishant Kheterpal, Jean-Baptiste Jeannin, Cyrus Omar. (2026). Hazel Prover: A Classroom Proof Assistant for Learning Structural Induction. https://arxiv.org/abs/2608.23309 doi:10.48550/arxiv.2608.23309
q2 · i? · design · r3
Cross-semester comparison of midterm induction-question results across six offerings of the class (Table 2 reports enrollments from 96 down to 16). The authors report the blank rate was "significantly higher in the first deployment than the second"; no effect size is printed.
"The clearest signal was the proportion of students who left the exam question blank. This was significantly higher in the first deployment than the second, suggesting the second tool at least led to much better confidence on pen-and-paper."
Discussion¶
Related Claims¶
- In the first deployment, exam scores on the induction question were visibly lower than in previous semesters taught without the tool — related
- After the design iteration, no student deleted an evaluation step, in contrast to frequent step deletion in the first deployment — related
- Student DP performance differed significantly across the three course deployments, with the paper-based DT3 exam scoring lowest, likely due to administration mode — related
- Student backtracking before setting up the correct induction decreased across exercises in both deployments, indicating a learning curve — related
- Classroom pretesting improves high-stakes exam performance on pretested content by 8–9% over a multi-week course — a broader claim this one bears on
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
manual-evaluation-steps-improve-exam-transfer |
title |
Requiring students to write out evaluation steps manually improved transfer to pen-and-paper exam performance between the two deployments |
description |
Requiring students to write out evaluation steps manually improved transfer to pen-and-paper exam performance between the two deployments |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — matthew-keenan-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.