Conventional reliability determination risks overestimating coding reliability in small group chat analysis¶

Claim · All claims
Evidence · 1 study · design r? · q2
Currency · newest study 2007 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? A risk for reliability overestimation was illustrated in the development of the multi-dimensional coding procedure for small group chat communication. → Jan Willem Strijbos 2007

Evidence¶

Jan Willem Strijbos 2007¶

Jan Willem Strijbos, Gerry Stahl. (2007). Methodological issues in developing a multi-dimensional coding procedure for small group chat communication. Learning and Instruction. https://hal.science/hal-00703042

q2 · i? · design · r?

Methodological illustration from the article's development of a multi-dimensional coding procedure: the authors demonstrate a case in which standard reliability determination would overestimate reliability, as one of the three issues the article addresses.

"In addition, a risk for reliability overestimation was illustrated."

Discussion¶

Related Claims¶

• Developing a multi-dimensional coding procedure for small group chat communication raises three methodological issues: unit of analysis and fragmentation, response-structure reconstruction, and reliability without overestimation — possibly the same claim (merge candidate)
• Threading — connections between analysis units — proved essential for handling unit fragmentation, reconstructing the response structure, and reliability of coding in small group chat — related
• In simulation, ignoring local dependence overestimates reliability, and the overestimation grows with the size of the testlet effect (17.0%-21.8% decrease under the larger testlet effect vs. 4.8%-7.4% under the small)