Bayes factor testing of homogeneous within-person variance in hierarchical models¶

Theory · All theories
Evidence · 2 claims (2 for) · 1 study (1 associational), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description¶

A methodological framework that reframes measurement reliability as an individual-level property rather than a stable property of a questionnaire or task. It uses "a novel approach, using the Bayes factor, wherein a researcher can directly test for homogeneous within-person variance in hierarchical models", addressing the descriptive nature of ICCs that "depends upon the assumption of a common within-person variance". The framework leads into "the notion of individually varying ICCs" and is demonstrated on cognitive inhibition tasks.

Design Implications¶

Context¶

Requirements¶

• Hierarchical models of repeated-measures data in which within-person variance can be parameterized and tested

Constraints¶

• Demonstrated in the article on cognitive inhibition tasks; the abstract presents ICCs as descriptive and assumption-dependent rather than invalid

Target Learners¶

• Researchers assessing measurement reliability in repeated-measure designs

Target Learning Objectives¶

• Testing whether reliability generalizes to the individual level
• Classifying individuals by within-person variance

Claims¶

• Heterogeneous Within Person Variance Cognitive Inhibition Tasks [+M]
• Few Individuals Common Variance Model Reliability Masking [+M]

Related Theories¶

• Membership model for classifying individuals into the common variance model
• Bayes factors as interpretable weight of evidence for competing scientific theories

Examples¶

• BFpack: an R package for Bayes factor hypothesis testing of common statistical testing problems
• vICC R package for Bayesian testing of within-person variance homogeneity

Key Sources¶

• Donald Williams, Stephen Martin, Phillipe Rast. (2021). Putting the individual into reliability: Bayesian testing of homogeneous within-person variance in hierarchical models. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01646-x