Students who experienced annotation subjectivity firsthand most frequently requested ways to reduce disagreement, indicating they had not internalised disagreement as meaningful signal¶

Claim · All claims
Evidence · 1 study · qualitative r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The most frequent improvement suggestion was to make annotations more consistent via clearer guidelines, calibration exercises, or reference images. → Ralf Raumanns 2026 q2 i? Students treated inconsistency as a solvable problem rather than recognising it as an inherent feature of the task, contrasting with the subjectivity-aware learning persona. → Ralf Raumanns 2026

Evidence¶

Ralf Raumanns 2026¶

Ralf Raumanns, Theresa Elstner, Louis Ferger-Andrews, Louise M. Carlsen, Martin Potthast, Gerard Schouten, Josien P. W. Pluim, and Veronika Cheplygina. (2026). Data annotations as pedagogical hints: from subjective labels to critical thinking. https://arxiv.org/abs/2607.20149 doi:10.48550/arxiv.2607.20149

q2 · i? · qualitative · r2

Thematic analysis (Braun and Clarke six-step framework, two-coder consensus, relaxed F1 0.71) of open-ended survey responses. Consistency-seeking was the top theme (n̄ = 20); students "requested that it be removed" after recognising ambiguity.

"The most frequent suggestion was to make an- notations more consistent, for example through clearer guidelines, calibration exercises, or refer- ence images ( ¯n = 20 )."

Discussion¶

Related Claims¶

• CLARA annotations align with blinded educator judgments, with 0.81 pairwise agreement and highest-rated educational interpretability — related
• In disagreement cases, blinded educators more frequently prefer CLARA developmental estimates over publisher recommendations — related
• In one experiment, students who generated their own underlining did not score significantly higher on tests than students given experimenter-generated underlining — related
• Human and agent framework revisions concentrate on the mid-level engagement categories (Levels 3–5), indicating greater ambiguity there than at scale endpoints — related