Switching from reference-based to contextual evaluation prompts improved GenAI grading accuracy¶
Claim · All claims
Evidence · 1 study · designr2·q1
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q1 i? A contextual evaluation prompt that scores responses solely on relevance and completeness to the question, without reference answers, markedly improved the models' ability to assess open-ended responses. → Pecuchova 2025
Evidence¶
Pecuchova 2025¶
Pecuchova, J., Benko, Ľ., & Drlik, M. (2025). Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00517-2
q1 · i? · design · r2
Modeling phase design iteration: after reference-based prompts underperformed, a contextual prompt evaluating answers "without relying on external references" was developed and improved assessment.
"The introduction of the contextual evaluation prompt marked a significant improvement in the model’s ability to accurately assess open-ended student responses."
Discussion¶
Related Claims¶
- Hint-based prompts raise ChatGPT scoring agreement with human experts to 98–100%, while no-hint prompts are unstable (Q2 precision 0.42) — related
- ChatGPT zero-shot relevance extraction for instruction-quality assessment retrieves mostly irrelevant utterances and overestimates instruction quality — related
- Phase VI AI-enabled systems face six documented challenges: explainability, hallucination risk, prompt sensitivity, computational cost, validation complexity, and limited large-scale evidence — a broader claim this one bears on
- Zero-shot LLM errors follow four recurring patterns: over-interpretation, failure to detect relevant information, hallucination, and failure to generate a response — related
- ChatGPT feedback required doctoral students to validate the procedural accuracy of algorithm-generated outputs, with judgement shaped by prompt quality — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
contextual-evaluation-prompt-improvement |
title |
Switching from reference-based to contextual evaluation prompts improved GenAI grading accuracy |
description |
Switching from reference-based to contextual evaluation prompts improved GenAI grading accuracy |
status |
review |
evidence_strength |
weak |
newest_evidence |
2025 |
recheck_by |
2028 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — pecuchova-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.