The chi-square significance test for NC-DIF is overly sensitive at large sample sizes, requiring an empirical cutoff value¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 1996 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 1999: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on an ERIC document (a report, conference paper or thesis). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? The chi-square test for NC-DIF was shown to be overly sensitive for large sample sizes, so an empirically established critical value (.016, the 99th percentile of 2,000 DIF-free items) was used instead. → Flowers 1996

Evidence¶

Flowers 1996¶

Flowers, C. P., Oshima, C., & Raju, N. (1996). A Description and Demonstration of the Polytomous-DFIT Framework. https://eric.ed.gov/?id=ED401319

q2 · i? · causal · r2

The article attributes the sensitivity finding to Fleer (1993). In this study, 2,000 DIF-free items were simulated and the alternative cutoff was set at the 99th percentile, "resulted in an alternativecutoff value of .016," applied to all DIF indices to protect against Type I error.

"This test was shownto be overly sensitivefor large sample sizes (Fleer, 1993).Fleer suggested empiricallyestablishing a critical (cutoff)value for NC-DIF."

Discussion¶

Related Claims¶

• The ICC and LDF methods identify substantially different sets of items as exhibiting DIF on a state science performance assessment — the general claim this finding bears on, with the studies behind it
• Grade cutoffs are arbitrary because adjacent scores receiving different grades are not significantly different, per the author's statistical argument — related
• Item discrimination moderates detection differently by index: NCDIF benefits from high a-parameters only at large DIF magnitude — related
• Lord's chi-square identified more DIF items than Raju's unsigned area in the without-purification condition — related
• C-DIF was less stable than NC-DIF across simulated conditions — related
• The polytomous-DFIT framework effectively identified DTF and DIF in polytomously scored data under the simulated conditions — related
• Raters differed significantly in severity/leniency in the MFRM analysis — related
• Test length, focal group distribution, number of DIF items, and direction of DIF had little effect on detection rates — related
• Item purification made both DIF methods more sensitive, flagging more items — related