Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2025 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Sentence embedding models achieved mostly moderate agreement with human graders and are not appropriate for open-ended assessment without further human supervision. → Pecuchova 2025
Evidence¶
Pecuchova 2025¶
Pecuchova, J., Benko, Ľ., & Drlik, M. (2025). Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00517-2
q2 · i? · design · r2
Evaluation of SBERT, USE, T5, BERT and RoBERTa using cosine similarity against reference answers in the same 1,885-response dataset; "Most of the sentence embedding models' agreement is at moderate level" with human graders.
"Most of the sentence embedding models’ agreement is at moderate level and would be not appropriate to use these models for open-ended questions assessment without further human supervision."
Discussion¶
Related Claims¶
- Even the best LLM grader reaches only moderate agreement with human teachers, below the threshold for good reliability — related
- Two human graders reached almost perfect agreement (QWK = 0.95), though one waterfall-model question showed only substantial agreement — related
- Fine-tuned PLMs measure high-inference teaching practices better when the variable requires less pedagogical expertise, matching human-rater agreement on lexical variables but degrading on inference-heavy ones — related
- GPT-4 achieved almost perfect inter-rater reliability with faculty after three iterations of rubric engineering on three of four medical school OEQ questions — related
- The most challenging holistic-rubric question reached only moderate inter-rater reliability (kw=0.54) after three iterations — related
- Most faculty–GPT-4 scoring discrepancies were only one-point differences, shrinking across iterations — related
- Comparison models (RoBERTa, SBERT, naïve GPT-4o prompting) failed to reach acceptable agreement on PCK items, with Cohen's κ at or below .55 — related
- LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — a broader claim this one bears on
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
embedding-models-moderate-agreement-need-supervision |
title |
Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading |
description |
Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2025 |
source_checked |
2026-10-20 |
recheck_by |
2028 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — pecuchova-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.