Response styles as scale-usage differences independent of true score¶

Theory · All theories
Evidence · 2 claims (2 for) · 1 study (1 associational), q1 · 0 of 1 report an effect size · 2 claims rest on one study

Description¶

Response styles are patterns in how survey respondents use Likert scale categories. The article defines them as occurring when "respondents use the categories of the Likert scale differently despite having the same true score on the construct of interest," giving extreme-category overuse as an example. The framework treats response styles as a construct-irrelevant source of bias in survey scores, which the study examines for its consequences for longitudinal growth estimation on social-emotional constructs.

Design Implications¶

Context¶

Requirements¶

• Longitudinal Likert-scale survey data on a psychological construct
• Scoring models that can either account for or ignore response styles

Constraints¶

• Effects on growth estimates vary by psychological construct, response style, and model used, so the framework's consequences are not uniform across measures

Target Learners¶

• K-12 students completing social-emotional learning surveys

Target Learning Objectives¶

• Accurate measurement of social-emotional construct growth over time

Claims¶

• Response Styles Affect Growth Estimates Sel [+M]
• Differing response styles create a construct-irrelevant source of bias that distorts fundamental inferences made from survey data [+W]

Related Theories¶

• Response styles as construct-irrelevant variation in Likert scale use

Examples¶

• Compare growth estimates from IRT scoring models that do and do not account for response styles when evaluating longitudinal survey scores
• Four years of longitudinal social-emotional survey scores analyzed under competing IRT scoring models

Key Sources¶

• James Soland, Megan Kuhfeld. (2020). Do response styles affect estimates of growth on social-emotional constructs? Evidence from four years of longitudinal survey scores. NWEA Research Working Paper. https://www.nwea.org/research/publication/do-response-styles-affect-estimates-of-growth-on-social-emotional-constructs-evidence-from-four-years-of-longitudinal-survey-scores/