Screen for local dependence with simple methods (Q3 statistic or item-based versus testlet-based reliability comparison) before fitting a complex testlet model¶

Strategy · All strategies
Evidence · 1 claim (1 for) · 1 study (1 design), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description¶

Because fitting the general polytomous testlet model is time-consuming, the report recommends first applying relatively simple methods to detect local dependence and measure the magnitude of the testlet effect. It states that "some relatively simple methods to detect local dependence and measure the magnitude of the testlet effect may be used before applying the model." For dichotomous items, Yen's Q3 statistic can be used; alternatively, one can compare item-based reliability with testlet-based reliability, the method used in this study, which the report notes "can be applied to both dichotomous and polytomous items."

Design Implications¶

Context¶

Requirements¶

• Item response data organized by testlet so that passage-based (testlet-summed) reliability can be computed alongside item-based reliability

Constraints¶

• The report notes the Q3 statistic applies to dichotomous items, while the reliability-comparison method applies to both dichotomous and polytomous items

Target Learners¶

• Examinees of testlet-based assessments in testing programs

Target Learning Goals¶

• Deciding whether local dependence modeling is warranted before investing in complex testlet model estimation

Claims¶

• In operational reading-test data, ignoring local dependence inflates reliability estimates, with passage-based alpha 2.3%-4.9% lower than item-based alpha across six test forms [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Yanmei Li, Shuhong Li, and Lin Wang. (2010). Application of a General Polytomous Testlet Model to the Reading Section of a Large-Scale English Language Assessment. ETS Research Report RR-10-21. http://www.ets.org/research/contact.html