Combine holistic with analytic scores as a complementary approach to enhance rating validity¶

Strategy · All strategies
Evidence · 3 claims (2 for, 1 mixed) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 3 claims rest on one study

Description¶

The article relays a recommendation, attributed by its authors to prior work, that rating validity can be improved by pairing grading modes rather than choosing one: "a way to enhance rating validity could be through the use of a complementary approach, which combines holistic with analytic scores". The article's own findings — slight analytic gains in expert consensus and low agreement under both approaches — motivate combination. The authors also note evidence that holistic marking practices could be improved by introducing analytic rubrics for feedback as an ancillary during marking.

Design Implications¶

Context¶

Requirements¶

• Raters must have access to both a holistic scale and an analytic rubric for the same student work, as participants in this study's grading task did.

Constraints¶

• The article reports the analytic-approach improvement in consistency over holistic as minimal in its own data.

Target Learners¶

• pre-service teachers
• in-service teachers rating student writing

Target Learning Goals¶

• reliable and valid grading of student writing
• assessment consistency in teacher preparation

Claims¶

• Analytic grades matched expert criteria more often for weak essays while holistic grades did so for strong essays [+W]
• Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality [+W]
• Pre-service teachers show high intra-rater consistency between holistic and analytic grades for the same essays [~W]

Related Strategies¶

• Combine inter-learner rankings with intra-learner diagnostics in assessment and feedback
• Pair convenient platform analytics with transparent, education-specific evaluation criteria rather than trusting either alone
• Match the refinement coordination strategy to the quality criterion: decentralized refinement for localized criteria, centralized planning for holistic ones

Examples¶

-

Key Sources¶

• Lopera-Oquendo, C., Lipnevich, A. A., & Mañez, I. (2024). Rating writing: Comparison of holistic and analytic grading approaches in pre-service teachers. Learning and Instruction, 94, 101992. https://doi.org/10.1016/j.learninstruc.2024.101992