The ICC and LDF methods identify substantially different sets of items as exhibiting DIF on a state science performance assessment¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 1998 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2001: tool and model accuracy changes quickly

Subclaims¶

q2 i? On a state-level performance assessment, only items 3, 4 and 5 were flagged by both methods; four items were unique to LDF and four unique to ICC. → Holweger 1998

Evidence¶

Holweger 1998¶

Holweger, Nancy; Weston, Timothy. (1998). Differential Item Functioning: An Applied Comparison of the Item Characteristic Curve Method with the Logistic Discriminant Function Method. https://eric.ed.gov/?id=ED422362

q2 · i? · associational · r2

Applied comparison on a state-level performance-based assessment with approximately 16,000 examinees (8,539 females, 8,029 males after removing zero scores). The authors report "there is no uniformity across techniques" in the seven-item lists each method produced, with only partial overlap.

"Items 3, 4 and 5 are identified by both techniques as having substantial DIF. However, items 12, 29, 13, and 24 show DIF using the LDF technique but not the ICC technique. Items 18, 23, 25 and 26 show DIF using the ICC technique but not the LDF technique."

Studies bearing on this claim¶

How far does a statistical item-bias (DIF) flag identify a biased item, rather than the method, sample or form that produced it? This table gathers the studies behind this claim and the 6 narrower claims that bear on it: 5 distinct studies (3 associational, 2 causal), q1–q2, 0 of 5 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Items flagged as DIF in booklet comparisons displayed non-uniform DIF in their item characteristic curves — related
• Seven items exhibit absolute gender DIF by the LDF method, flagged when the total-score line falls outside .95 confidence bands — related
• Given assumption violations and the uniformity factor, the authors judge the LDF method more likely accurate and superior to ICC for this performance assessment — related
• ICC method flags seven items with the largest DIF indices, two of which show slight nonuniform DIF that the unsigned-area measure understates — related
• The assessment violates IRT assumptions of unidimensionality and local independence, which the authors identify as a likely cause of the method discrepancy — related
• Some 2003 MCAS fourth-grade math word problems exhibited differential item functioning for English-language learners versus non-ELLs — related
• DIF was found for four items but could not be anticipated from differential response times between boys and girls alone — related