The assessment violates IRT assumptions of unidimensionality and local independence, which the authors identify as a likely cause of the method discrepancy¶

Claim · All claims
Evidence · 1 study · associational r2 · q1
Currency · newest study 1998 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2001: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on an ERIC document (a report, conference paper or thesis). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q1 i? The assessment measures multiple constructs (science plus reading and math items, six domains) and groups related items (soil tests, levers, salinity), violating unidimensionality and local independence. → Holweger 1998

Evidence¶

Holweger 1998¶

Holweger, Nancy; Weston, Timothy. (1998). Differential Item Functioning: An Applied Comparison of the Item Characteristic Curve Method with the Logistic Discriminant Function Method. https://eric.ed.gov/?id=ED422362

q1 · i? · associational · r2

Authors' diagnostic discussion (type e): five items assess science and reading concurrently, seven science and math, and items cluster into soil-test, lever, and salinity content sets, so related items "cannot actually be considered separate items in the item response theory sense." They conclude the ICC method's errors are more likely affected than LDF's.

"The most glaring problem with this assessment is that it most likely measures more than one construct,thus violating the assumption of unidimensionality."

Discussion¶

Related Claims¶

• Achievement dimensionality appears to change from pretest to end-of-course testing, questioning unidimensional ICC measurement of individual growth during instruction — related
• In an empirical comparison on a testlet-based English assessment test, the bifactor model is preferred over the second-order and unidimensional 2PL models by both AIC and BIC — related
• The ICC and LDF methods identify substantially different sets of items as exhibiting DIF on a state science performance assessment — related
• ICC method flags seven items with the largest DIF indices, two of which show slight nonuniform DIF that the unsigned-area measure understates — related
• Ignoring testlet effects by fitting a unidimensional 2PL model produced mild shrinkage of item intercept and loading estimates toward zero relative to the bifactor model — related
• Given assumption violations and the uniformity factor, the authors judge the LDF method more likely accurate and superior to ICC for this performance assessment — related
• Most Turkish IRT-based scale development studies did not check the unidimensionality or local independence assumptions — related
• The multivariate normality assumption of the true score model was judged reasonable for all domains and grades — related
• A multidimensional IRT model with simple structure fits testlet-based reading data better than both the general polytomous testlet model and the standard 2PL/GPCM — related