Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades¶

Claim · All claims
Evidence · 1 study (2 entries) · review r3 · q2–q3
Currency · newest study 2026 · checked against its sources 2026-10-16 (automated, 2 of 2 entries) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Testing prompts beforehand and using detailed scoring rubrics supplemented with sample correct answers is especially vital for consistent and effective grading. → Marcin Jukiewicz and Michał Wyrwa 2026 q2 i? In one reviewed study, using a detailed rubric improved AI grading accuracy by 9% and F1 by 15% compared to classical general scoring. → Marcin Jukiewicz and Michał Wyrwa 2026 (2)

Evidence¶

Marcin Jukiewicz and Michał Wyrwa 2026¶

Marcin Jukiewicz and Michał Wyrwa. (2026). Can ChatGPT Replace the Teacher in Assessment? A Review of Research on the Use of Large Language Models in Grading and Providing Feedback. Appl. Sci. 2026, 16, 680. https://doi.org/10.3390/app16020680

q3 · i? · review · r3

The review's RQ4 synthesis identifies prompting as the heaviest influence on grading outcomes, with rubrics and exemplar answers named as vital for consistency. No effect size is printed for this synthesis-level statement.

"The final outcome is especially heavily influenced by the prompt given to the GenAI. Testing prompts beforehand and focusing on their precision, for instance by having detailed scoring rubrics supplemented with sample correct answers, seem to be especially vital for consistent and effective grading"

Marcin Jukiewicz and Michał Wyrwa 2026 (2)¶

Marcin Jukiewicz and Michał Wyrwa. (2026). Can ChatGPT Replace the Teacher in Assessment? A Review of Research on the Use of Large Language Models in Grading and Providing Feedback. Appl. Sci. 2026, 16, 680. https://doi.org/10.3390/app16020680

q2 · i? · review · r3

The review reports, from a single chemistry study in its table, that detailed rubrics improved accuracy by 9% and F1 by 15% over general scoring. These are printed percentage improvements, not standardized effect sizes.

"Using a detailed rubric (i.e., breaking grading into many small criteria) improves AI's results by 9% (accuracy) and 15% (F1) compared to classical, general scoring."

Studies bearing on this claim¶

When an AI model scores work against a rubric, does the rubric's wording and detail decide how closely it agrees with human markers? This table gathers the studies behind this claim and the 8 narrower claims that bear on it: 8 distinct studies (4 causal, 2 associational, 1 review, 1 design), q2–q3, 0 of 8 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
• Model performance is robust to minor prompt wording changes but sensitive to holistic rubric redesign — related
• AI scoring consistency approaches or exceeds 90% for most rubric items, but spelling is replaced by a deterministic LanguageTool-based approach achieving 68% accuracy with 100% consistency — related
• Prompt design significantly influences LLM accuracy on IVDA MCQs, with slide context reducing smaller-model accuracy but improving GPT models, while persona effects are inconsistent — related
• LLM accuracy increases monotonically with prompt sophistication across semantically equivalent medical questions, with Low Literacy users scoring significantly below Prompt Engineer users — related
• Curricular chain-of-thought prompting improves competency-classification accuracy over zero-shot, with gains concentrated in larger models, while definition-based prompting does not improve performance — related
• Hint-based prompts raise ChatGPT scoring agreement with human experts to 98–100%, while no-hint prompts are unstable (Q2 precision 0.42) — a narrower finding that bears on this claim
• Removing detailed rubric explanations from prompts improved grading accuracy across all approaches in an ablation study — reports the opposite
• GPT models achieved high polarity agreement with instructors only for positive feedback on the Topic aspect, and could not reliably assess the other rubric aspects — related
• Rubric-enhanced prompting (Variant 2) raises LLM assigned scores relative to the no-rubric baseline, with model-specific gaps at particular taxonomy levels — related
• A construction-focused correctness rubric underperformed behavior-focused and undefined prompts for LLM scoring of code explanations — reports the opposite
• Expert-in-the-loop revision substantially improves medical accuracy and educational quality of generated storylines — related
• LLMs overrate short underdeveloped essays and over-penalize minor language errors in well-developed essays — related
• LLMs support the shift from memorization-focused to competency-based assessment by generating rubrics, varied assessment items and authentic scenario-based tasks — related