Perceptions of feedback characteristics vary between researcher and teacher coders, with agreement shaped by professional background¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Combined coder agreement was weak (kappa below 0.30) for context-dependent codes such as Specific, Learning Strategy Suggestion, and Hints, while Emoji and Positive Affect showed strong agreement among all coders. → Seiyon M. Lee 2025 q2 i? Researchers and teachers showed distinct correlation patterns, suggesting each group's interpretative lens is shaped by professional backgrounds and experiences. → Seiyon M. Lee 2025 (2)

Evidence¶

Seiyon M. Lee 2025¶

Seiyon M. Lee, Sami Baral, Hongming Chip Li, Li Cheng, Shan Zhang, Carly S. Thorp, Jennifer St. John, Tamisha Thompson, Neil Heffernan, Anthony F. Botelho. (2025). Developing Feedback Taxonomy for Math: A Synergy of Perspectives through Data Mining Methods. Journal of Educational Data Mining, Volume 17, No 2. https://osf.io/puhyr/?view_only=514d8534e9be41d9a57ad82e9efb3249

q2 · i? · design · r2

Fleiss' kappa computed over six coders (three researchers, three mathematics teachers with over 20 years of experience) coding 100 feedback messages. The article reports weak combined agreement "below 0.30" for the named codes and strong agreement for Emoji and Positive Affect; no effect size is printed.

"while the research team generally showed higher agreement than the teacher experts, we found weak agreement with combined kappa values below 0.30 for certain codes, such as "Specific," "Learning Strategy Suggestion," and "Hints.""

Seiyon M. Lee 2025 (2)¶

Seiyon M. Lee, Sami Baral, Hongming Chip Li, Li Cheng, Shan Zhang, Carly S. Thorp, Jennifer St. John, Tamisha Thompson, Neil Heffernan, Anthony F. Botelho. (2025). Developing Feedback Taxonomy for Math: A Synergy of Perspectives through Data Mining Methods. Journal of Educational Data Mining, Volume 17, No 2. https://osf.io/puhyr/?view_only=514d8534e9be41d9a57ad82e9efb3249

q2 · i? · qualitative · r3

Spearman's rank correlation analysis of coding by researchers versus teachers on the 100-message sample. The article reports group-specific correlation patterns, e.g. researchers linking Specific with Correction and Explanation, teachers linking Mathematical Suggestions with Specific and Explanation; no effect size printed.

"Overall, the findings suggest that each group's interpretative lens is shaped by their professional backgrounds and experiences. For researchers, the emphasis was on the diagnostic function of feedback, while for teachers, it centered on pedagogical design underlying the feedback."

Discussion¶

Related Claims¶

• Exploratory factor analysis of coder-assigned feedback codes yields 12 latent factors, several combining multiple codes with inverse loadings — related
• Constructs with higher operational clarity show higher overall coder agreement, and low clarity harms human coder agreement more than LLM agreement — related
• Some teacher feedback codes commonly co-occur, such as Confirmation with Positive Affect and Mathematical Suggestions with Hints — related
• GenAI-supported peer feedback groups produced more suggestions and explanations and less positive affective feedback than the PF group — related
• A general-purpose LLM assessing team emails' emotional tone was biased toward interpreting messages as anxiety-related only — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• Prior non-LLM automated coding approaches yielded variable and often lower agreement with human coders on teacher knowledge measures, especially PCK — related
• Coder agreement ranged from 68.57% (identical) to 95% (category coding) across coding stages — related