Cross-validation-selected hyperparameters generalized to the test set only for Situation and Clarity, with the other four dimensions showing CV–test misalignment¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? CV-selected configurations matched best test performance only for Situation and Clarity; remaining dimensions showed misalignment attributed to severe class imbalance and limited training samples. → Firdausi 2026
Evidence¶
Firdausi 2026¶
Firdausi, H.; Wasis; Sholikah, R. W.; Lemantara, J.; Rosyadi, F. D.; Ginardi, R. V. H.; Pradityo, M. H. A.; Hakim, A. R. (2026). Educational Innovation through Automated Essay Scoring: A Multidimensional Framework for Evaluating Critical Thinking in High School Physics Essays. International Journal of Intelligent Engineering and Systems, Vol.19, No.8. https://doi.org/10.22266/ijies2026.0831.09
q2 · i? · design · r2
Comparison of stratified 5-fold CV results (Table 9) with held-out test results (Table 10) across six dimensions in Results §4.3. The article reports CV QWK varied widely, e.g., Inference 0.118 ± 0.185 with high variance.
"Notably, Situation and Clarity were the only two dimensions in which the CV-selected hyperparameters generalized consistently to the test set, whereas the remaining dimensions exhibited misalignment between CV estimates and test performance, likely attributable to severe class imbalance and limited training samples"
Discussion¶
Related Claims¶
- The best AES models per FRISCO dimension reached strong agreement for Situation (QWK 0.728) and Clarity (QWK 0.763) but only fair agreement for Focus, Reason, and Inference in Indonesian physics essays — related
- Decision quality collapses toward the majority action in severely imbalanced settings — related
- Core-Ed dimensions capture part of the FineWeb-Edu scalar signal while remaining not individually redundant with it (six-dimension cross-validated R2 of 0.228) — related
- Core-Ed Edu-QuRater scores vary sensibly with education-level metadata on out-of-sample materials (marginal Pearson correlations 0.62 and 0.64; six-dimension cross-validated R2 of 0.449) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
cv-test-misalignment-small-imbalanced-data |
title |
Cross-validation-selected hyperparameters generalized to the test set only for Situation and Clarity, with the other four dimensions showing CV–test misalignment |
description |
Cross-validation-selected hyperparameters generalized to the test set only for Situation and Clarity, with the other four dimensions showing CV–test misalignment |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — firdausi-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.