A heuristic for assigning blame to rules behind incorrect steps agreed with human coders at kappa=0.78¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2015 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Nwaigwe's best heuristic for assigning blame to rules behind incorrect steps agreed well with human coders, with kappa=0.78, and was adopted in Andes3. → VanLehn 2015

Evidence¶

VanLehn 2015¶

VanLehn, K. (2015). Reflections on Andes' Goal-free User Interface. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-015-0067-7

q2 · i? · review · r2

Adaeze Nwaigwe developed a heuristic solution to the assignment-of-blame problem to support educational data mining. The article reports the best heuristic "agreed well (kappa=0.78) with human coders" and that Andes3 adopted it, blaming rules via small edits, user corrections, or replacement steps of the same type.

"She studied several heuristics, and found that the best one agreed well (kappa=0.78) with human coders (Nwaigwe et al. 2007)."

Discussion¶

Related Claims¶

• Automatic DAC coding of dialogue acts agrees substantially with manual coding (79.0% agreement, kappa 0.75) — related
• AI assessors show emerging reliability: acceptable AI-human rater agreement (Cohen's 0.78) in a 12-educator case study and human-comparable feedback helpfulness ratings with 45 learners — related
• The emergent LLM coding pipeline reaches substantial agreement with expert human coding (Cohen's κ = 0.78, raw agreement 84%) on a stratified 10% sample — related