Type I error rates of all three polytomous DIF indices increase as the number of DIF items and DIF magnitude increase¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 1997 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2000: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on an ERIC document (a report, conference paper or thesis). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? Across NCDIF, extended SIBTEST, and extended Lord's chi-square, simulated Type I error rates rose with more embedded DIF items and larger DIF magnitude. → Flowers 1997

Evidence¶

Flowers 1997¶

Flowers, C. P., Oshima, T. C., & Raju, N. S. (1997). The Relationship between Polytomous DFIT and Other Polytomous DIF Procedures. Paper presented at the NCME Annual Meeting, Chicago. https://eric.ed.gov/?id=ED410300

q2 · i? · causal · r2

Simulation results across conditions varying two versus four DIF items and .10 versus .25 DIF magnitude, reported in Table 2. The authors present this as an expected pattern rather than a tested hypothesis.

"All indices showed an increase error rate as the number of DIF items and the magnitude of the DIF increased. As expected, the condition with the most DIF items and the greatest magnitude had the largest Type I error rate."

Discussion¶

Related Claims¶

• The ICC and LDF methods identify substantially different sets of items as exhibiting DIF on a state science performance assessment — the general claim this finding bears on, with the studies behind it
• The polytomous DFIT framework shows Type I error rates close to nominal alpha except when the number of DIF items and DIF magnitude are highest — related
• DIF detection rates for all indices are higher with larger samples, equivalent distributions, fewer DIF items, greater DIF magnitude, and larger a-parameters — related
• Test length, focal group distribution, number of DIF items, and direction of DIF had little effect on detection rates — related
• The polytomous-DFIT framework effectively identified DTF and DIF in polytomously scored data under the simulated conditions — related
• Item discrimination moderates detection differently by index: NCDIF benefits from high a-parameters only at large DIF magnitude — related
• Some items positioned close together across booklets were also flagged as DIF, possibly due to Type I error from sampling — related