Graduated assessment rubric separating answer correctness from method validity¶
Research Method · All research methods
Evidence · 3 claims (2 for, 1 mixed) · 1 study (1 causal),q2· 0 of 1 report an effect size · 3 claims rest on one study
Description¶
The rubric adapts the mathematical assessment distinction between the answer mark (correct output) and the method mark (valid reasoning) to automated diagnosis. For correct-answer cases it distinguishes four categories: clear reasoning, needs clarification, misconception detected, and wrong answer. The paper argues a binary formulation "collapses Options 2 and 3, either falsely accusing students whose explanations are merely brief, or losing the routing signal that distinguishes unclear from flawed reasoning." It grounds the design in Black and Wiliam's formative assessment steps.
Accounts¶
- Graduated assessment rubric separating answer correctness from method validity: The rubric adapts the mathematical assessment distinction between the answer mark (correct output) and the method mark (valid reasoning) to automated diagnosis. For correct-answer cases it distinguishes four categories: clear reasoning, needs clarification, misconception detected, and wrong answer. The paper argues a binary formulation "collapses Options 2 and 3, either falsely accusing students whose explanations are merely brief, or losing the routing signal that distinguishes unclear from flawed reasoning." It grounds the design in Black and Wiliam's formative assessment steps. (Moiz Imran (2026))
- Three-way KG-grounded diagnosis of student proof steps (optimal, valid-alternative, incorrect): The article's organizing evaluation framework classifies each student next step into three pedagogically distinct categories using a knowledge graph encoding all valid inference paths for each problem. The article states that "Unlike prior benchmarks relying on binary correctness, our KG encodes all valid inference paths, enabling three-way diagnosis." A step is Optimal if it follows an expert-authored path and strictly reduces distance to the conclusion, Valid-alternative if it is a valid inference that does not reduce distance, and Incorrect if it is not derivable under any valid rule. Edge-level annotation of rules and parent statements enables verification of both what was derived and how the derivation is justified. (Tahreem Yasir et al. (2026))
Claims¶
- LLM feedback agents classify optimal logic proof steps near-ceiling but perform poorly on valid-alternative and incorrect solutions [+M]
- Model selection, not feedback condition, explains LLM diagnostic accuracy in three-way logic proof classification [+M]
- LLM feedback agents show model-specific over-rejection of valid-alternative reasoning and over-validation of incorrect solutions [~W]
Related Research Methods¶
-
Key Sources¶
- Moiz Imran, Sahan Bulathwela. (2026). The Correct Answer Trap: Pedagogically-Grounded Detection and Feedback for Hidden Misconceptions. PEAF 2026: Workshop on Pedagogical Evaluation of Automated Feedback. https://orcid.org/0000-0002-5878-2143
- Tahreem Yasir, Wenbo Li, Sam Gilson, Sutapa Dey Tithi, Xiaoyi Tian, Tiffany Barnes. (2026). Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most. arXiv:2605.16207. https://arxiv.org/abs/2605.16207
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
graduated-assessment-rubric-separating-answer-correctness-from-method-validity |
title |
Graduated assessment rubric separating answer correctness from method validity |
description |
The rubric adapts the mathematical assessment distinction between the answer mark (correct output) and the method mark (valid reasoning) to automated diagnosis. |
status |
review |
generated |
by process:settle-candidates on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.