The LoU Interview shows high interrater reliability, and the rating procedure was streamlined over two years without loss of agreement¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 1977 · not yet checked against its sources · not reviewed by a person
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on an ERIC document (a report, conference paper or thesis). It is kept because 2 design pages cite it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? In year two, a streamlined two-rater procedure yielded interrater reliability of .96 with 73% agreement, and interviewer-only ratings later agreed with final ratings 87% of the time. → Loucks 1977

Evidence¶

Loucks 1977¶

Loucks, Susan F. (1977). Levels of Use of the Innovation: The Conceptualization and Measurement of a Variable Useful for Assessing Innovation Implementation by Individuals. https://eric.ed.gov/?id=ED137947

q2 · i? · design · r3

Second-year (Fall 1975-Spring 1976) rating procedure in which the interviewer's immediate rating replaced one of two independent ratings to reduce cost. The article also reports interviewer ratings had "a high percent agreement (87%) with final ratings", leading to interviewer-only rating with periodic reliability checks.

"Using this procedure, interrater reliability on overall LoU rating was .96, with 73% agreement between the two raters."

Discussion¶

Related Claims¶

• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
• LoU Interview ratings correlated .65 with readers' ratings of written ethnographic protocols, which the author read as support for the interview's validity tempered by second-hand information — related
• Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making it unsuitable for pass/fail decision consistency — related
• The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
• The interview coding achieved high inter-rater reliability, with Cohen's Kappa of 0.82 between independent coders — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• The review's coding framework showed very good inter-coder agreement (κ = 0.89, p < 0.001) on a random sample of six papers — related
• In Study 2, inter-rater agreement for rating creative problem solving ideas was good to excellent across indicators — related