Combine inter-learner rankings with intra-learner diagnostics in assessment and feedback¶

Strategy · All strategies
Evidence · 2 claims (2 for) · 1 study (1 design), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description¶

The authors' forward-looking recommendation for deploying analytic assessment: while inter-learner rankings provide essential comparative data, they should be paired with the self-referential, within-learner diagnostics that identify relative strengths and weaknesses in a learner's own profile. "This hybrid approach may offer a pathway to delivering more actionable and informative assessment and feedback to learners, teachers, and testers." The strategy follows directly from the finding that rank-based metrics alone can mask a system's diagnostic behaviour.

Design Implications¶

Context¶

Requirements¶

• Access to both inter-learner ranking data and calibrated analytic scores that support within-learner profile comparison

Constraints¶

• The authors state the hybrid approach may offer a pathway, framing it as a proposal rather than a tested result

Target Learners¶

• L2 English learners

Target Learning Goals¶

• Actionable diagnostic feedback on L2 writing proficiency

Claims¶

• Human operational raters remain stronger than LLMs at identifying relative strengths (positive feedback) within learner profiles [+W]
• In a self-referential framework, LLMs, particularly GPT-4.1, tend to outperform a single operational human rater at identifying relative weaknesses (negative feedback) across several Language and Content aspects [+W]

Related Strategies¶

• Combine holistic with analytic scores as a complementary approach to enhance rating validity

Examples¶

-

Key Sources¶

• Stefano Bannò, Kate Knill, Mark Gales. (2026). Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs. https://arxiv.org/abs/2605.04298