Three education experts rate PsyScore feedback significantly higher than all LLM baselines on every dimension (p < .001), with the No-IRT ablation declining in Adaptivity and Accuracy (p < .05)¶
Claim · All claims
Evidence · 1 study · designr2·q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? In a double-blind expert study of 80 essays, PsyScore outperforms GPT-4o, Llama-4 and Qwen-3 on all five dimensions (p < .001, paired t-test with Bonferroni correction); the No-IRT variant shows marked declines in Adaptivity and Accuracy (p < .05). → Wei Xia 2026
Evidence¶
Wei Xia 2026¶
Wei Xia, Jin Wu, Haoran Shi, Xiangyu Wang, Chanjin Zheng. (2026). PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback. arXiv preprint. https://arxiv.org/abs/2606.20287 doi:10.48550/arxiv.2606.20287
q3 · i? · design · r2
Human expert evaluation: three senior education experts rated a stratified sample of 80 essays with feedback on five dimensions (1–5 Likert); PsyScore mean ratings (e.g., Overall 4.25±.63) exceed baselines, with Fleiss' κ of 0.733 for Specificity.
"PsyScore significantly outperforms all LLM baselines (GPT-4o, Llama-4, Qwen-3) on every dimension ( p < .001, paired t-test with Bonferroni correction). The ablation variant (PsyScore-No-IRT) exhibits marked declines in Adaptivity and Accuracy( p < .05)"
Discussion¶
Related Claims¶
- In pairwise LLM-judge comparisons PsyScore-AEF wins over 90% in Actionability against GPT-4o and Llama, and removing the psychometric module reduces Adaptivity and Tone wins with PsyScore retaining above 75% — possibly the same claim (merge candidate)
- Removing grid-search initialization of IRT discrimination and difficulty parameters drops scoring performance from QWK 0.747 to 0.705 — related
- Ablating expert-informed structural components lowered LLM-judge scores on all dimensions, with largest drops in challenge quality, intent alignment, and domain grounding — related
- At trait level PsyScore-AES ranks first in 10 of 11 dimensions (mean QWK 0.735), improving Voice scoring from 0.619 to 0.740 while trailing T-MES on Style — related
- PsyScore-AES achieves a state-of-the-art average QWK of 0.747 on ASAP++, surpassing the strongest baseline SaMRL-large (0.722) and ranking first on 6 of 8 prompts — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
psyscore-expert-evaluation-outperforms-baselines |
title |
Three education experts rate PsyScore feedback significantly higher than all LLM baselines on every dimension (p < .001), with the No-IRT ablation declining in Adaptivity and Accuracy (p < .05) |
description |
Three education experts rate PsyScore feedback significantly higher than all LLM baselines on every dimension (p < .001), with the No-IRT ablation declining in Adaptivity and Accuracy (p < .05) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — wei-xia-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.