Experts rated KITE's feedback positively on scaffolding, guidance, coherence, and tone in 93.18% of judgments, with high interrater agreement (κ = 0.88)¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Expert rubric evaluation of 44 simulated student–KITE interaction triples yielded 93.18% Yes judgments on scaffolding, guidance, coherence, and tone, with Cohen's κ = 0.88 between annotators. → Mragisha Jain 2026 q2 i? Mistake remediation received 63.63% Yes judgments, with 29.55% marked N/A because the initial response was already correct. → Mragisha Jain 2026 (2)

Evidence¶

Mragisha Jain 2026¶

Mragisha Jain, Tirth Bhatt, Griffin Pitts, Aum Pandya, Peter Brusilovsky, Narges Norouzi, Arto Hellas, Juho Leinonen, Bita Akram. (2026). Retrieval-Augmented Tutoring for Algorithm Tracing and Problem-Solving in AI Education. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), co-located with ACL 2026. https://arxiv.org/abs/2605.12988 doi:10.48550/arxiv.2605.12988

q2 · i? · design · r2

Expert evaluation of 44 interaction triples (Round 1 answer, KITE feedback, Round 2 answer) using a three-dimension rubric adapted from prior work. The results section reports "93.18% Yes judgments on each dimension".

"KITE receives consistently high ratings for scaffolding, guidance, coherence, and tone, with 93.18% Yes judgments on each dimension."

Mragisha Jain 2026 (2)¶

Mragisha Jain, Tirth Bhatt, Griffin Pitts, Aum Pandya, Peter Brusilovsky, Narges Norouzi, Arto Hellas, Juho Leinonen, Bita Akram. (2026). Retrieval-Augmented Tutoring for Algorithm Tracing and Problem-Solving in AI Education. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), co-located with ACL 2026. https://arxiv.org/abs/2605.12988

q2 · i? · design · r2

Same expert rubric evaluation; the mistake remediation dimension was not applicable in 29.55% of cases. The article reports "63.63% Yes judgments" for mistake remediation.

"Mistake remediation receives 63.63% Yes judgments, but 29.55% of cases are marked N/A because the simulated student's initial response was already correct and no error identification was required."

Discussion¶

Related Claims¶

• KITE's feedback led simulated students to improve revised answers in 24 of 27 non-already-correct interactions (88.89%) — related
• The most common answer transition after KITE feedback was Partially Correct to Correct, occurring in 14 of 44 cases (31.82%) — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• A Gemini-based creativity autorater scores real students' complex multimedia creativity tasks on par with human experts (item Kappa 0.66; total-score Pearson r = 0.88) — related