The rule-based recipe annotation system reaches near-perfect inter-annotator agreement (Cohen's Kappa 0.85) and average accuracy 0.95 in a validation on sampled recipe segments.¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person
Design limits · Mejia-Domenzain 2025: no comparison group (read from the study by two models, 2026-10-27)

Subclaims¶

q2 i? Manual validation of the regular-expression annotation rules against human labels yielded Cohen's Kappa of 0.85 and average accuracy of 0.95. → Mejia-Domenzain 2025

Evidence¶

Mejia-Domenzain 2025¶

Mejia-Domenzain, P., Frej, J., Neshaei, S. P., Mouchel, L., Nazaretsky, T., Wambsganss, T., Bosselut, A., & Käser, T. (2025). Enhancing Procedural Writing Through Personalized Example Retrieval: A Case Study on Cooking Recipes. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-024-00405-1

q2 · i? · causal · r2

Evaluation on a random sample of recipe segments: two author-annotators labeled shuffled segments as present/missing per suggestion. The printed result is that "the average accuracy was 0 .95" with near-perfect agreement. No effect size is printed.

"The Cohen's Kappa score between the annotators was 0.85 (near perfect agreement (Landis & Koch, 1977)) and the average accuracy was 0 .95."

Discussion¶

Related Claims¶

• The star-to-sentiment labeling rule agrees with human review-level labels at Cohen's kappa 0.66 with 82% accuracy on 3,000 RateMyProfessor teacher reviews — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
• NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — a broader claim this one bears on
• In classroom video labeling, high raw agreement (often exceeding 90%) co-occurred with low chance-corrected κ due to temporal misalignment — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Automatic DAC coding of dialogue acts agrees substantially with manual coding (79.0% agreement, kappa 0.75) — related
• Gemini 2.0 Flash validated as a reliable LLM annotator of feedback focus types — related
• Machine learning detects five of eight reflective-writing categories with substantial or almost perfect reliability and three with moderate reliability (Cohen's κ .53–.85) — related