Improve LLM grading alignment through LLM-optimized guidelines, calibration examples, and median-of-parallels grading¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends concrete steps for future LLM-assisted grading: writing grading guidelines that are easier for LLMs to comprehend with no prior course knowledge, converting qualitative qualities into observable evidence requirements, asking the LLM to cite evidence as short quotes, calibrating with artificial top-, middle-, and low-grade examples, and grading in three rotated parallels to take a reasonable median. It states that "Three parallels of each assessment could ensure a reasonable median."

Design Implications¶

Context¶

Requirements¶

• Grading guidelines rewritten for LLM comprehension with no prior course knowledge
• A few example top-, middle-, and low-grade submissions for LLM calibration
• Multiple grading parallels with rotated batch positioning

Constraints¶

• One prior study reported a minimal effect of nuanced prompt optimization on essay-style grading, suggesting strong reliance on generic characteristics
• The study's guidelines were not written with LLMs in mind and lacked detailed grade mapping or rubrics

Target Learners¶

• master's-level public health and health sciences students

Target Learning Goals¶

• improved alignment of LLM-suggested grades with human examiner grades on essay-style exams

Related Strategies¶

• Design guidelines for future LLM evaluation tools: configurable metric views, tighter metric-rubric integration, and comparison-foregrounding visualizations
• Extend educational LLM benchmarks with broader item formats, human-in-the-loop item curation, and multi-turn evidence-based feedback measures

Examples¶

-

Key Sources¶

• Brevik A, Jerpseth H and Lafontan SR (2026). Large language models as grading assistants in public health education: a method-comparison study of essay-style exam assessment. Front. Educ. 11:1904452. https://doi.org/10.3389/feduc.2026.1904452