GLAT shows robust structural validity and reliability under a 2-parameter logistic IRT model¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? The 2PL model fit better than the Rasch model, while the 3PL model did not significantly improve fit over the 2PL model. → Yueqiao Jin 2024 q2 i? The final 2PL model showed good-to-excellent fit indices and acceptable internal consistency. → Yueqiao Jin 2024 (2)

Evidence¶

Yueqiao Jin 2024¶

Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević and Lixiang Yan. (2024). GLAT: The Generative AI Literacy Assessment Test. https://arxiv.org/abs/2411.00283 doi:10.48550/arxiv.2411.00283

q2 · i? · design · r2

Likelihood-ratio tests compared nested IRT models on the 355-student sample. The article reports "asignificantimprovement in fit with the 2PL model" (χ2 = 55.596, df = 19, p < 0.001); the 3PL did not improve on the 2PL (χ2 = 23.888, df = 20, p = 0.247).

"theANOVAresultsindicatedasignificantimprovement in fit with the 2PL model𝜒 2 = 55.596,𝑑𝑓 = 19,𝑝 < 0.001"

Yueqiao Jin 2024 (2)¶

Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević and Lixiang Yan. (2024). GLAT: The Generative AI Literacy Assessment Test. https://arxiv.org/abs/2411.00283

q2 · i? · design · r2

The article reports the final 2PL model's fit (M2 = 225.325, RMSEA 0.031, SRMSR 0.052, TLI 0.967, CFI 0.970) and reliability: "Cronbach's alpha = 0.80; omega total = 0.81", both above the 0.7 acceptability threshold.

"resulting in a reliable 2-parameter logistic (2PL) model (Cronbach’s alpha = 0.80; omega total = 0.81) with a robust factor structure (RMSEA = 0.03; CFI = 0.97)"

Discussion¶

Related Claims¶

• Ignoring testlet effects by fitting a unidimensional 2PL model produced mild shrinkage of item intercept and loading estimates toward zero relative to the bifactor model — related
• A multidimensional IRT model with simple structure fits testlet-based reading data better than both the general polytomous testlet model and the standard 2PL/GPCM — related
• In an empirical comparison on a testlet-based English assessment test, the bifactor model is preferred over the second-order and unidimensional 2PL models by both AIC and BIC — related
• Some reviewed studies with polytomous items used dichotomous IRT models such as 2PLM or Rasch — related
• In operational reading-test data, item parameter estimates from a standard 2PL/GPCM that ignores local dependence closely match those from the testlet model (average correlations 0.9893 and 0.9978) — related
• Item parceling at the testlet level substantially improves model fit, allowing partial (Rasch, 2PL) or full (bifactor) recovery of CAT constructs — related
• GLAT scores predicted performance on GenAI-supported tasks better than perceived ChatGPT proficiency — related
• Most Turkish IRT-based scale development studies did not check the unidimensionality or local independence assumptions — related