CTT item selection dropped five low-discrimination items, retaining 20 items of moderate difficulty¶

Claim · All claims
Evidence · 1 study (2 entries) · design r3 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Five items with discrimination indices below 0.3 were excluded from the GLAT. → Yueqiao Jin 2024 q2 i? Retained items showed discrimination indices from 0.33 to 0.55 and difficulties from 0.25 to 0.90. → Yueqiao Jin 2024 (2)

Evidence¶

Yueqiao Jin 2024¶

Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević and Lixiang Yan. (2024). GLAT: The Generative AI Literacy Assessment Test. https://arxiv.org/abs/2411.00283 doi:10.48550/arxiv.2411.00283

q2 · i? · design · r3

Classical test theory item analysis with 200 higher education students computed item difficulty and point-biserial discrimination; items 6, 9, 14, 15, and 20 fell below the 0.3 discrimination threshold and were excluded.

"atotaloffiveitems(Items6,9,14,15,and20)haddiscriminationindicesbelowthethresholdof0.3."

Yueqiao Jin 2024 (2)¶

Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević and Lixiang Yan. (2024). GLAT: The Generative AI Literacy Assessment Test. https://arxiv.org/abs/2411.00283

q2 · i? · design · r3

The article reports the retained items' difficulty spread and that discrimination indices ranged from 0.33 to 0.55 (M = 0.41, SD = 0.06), indicating items differentiate high and low performers across proficiency levels.

"Of the 20 items retained, the item difficulties ranged between 0.25 and 0.90 (M = 0.67, SD = 0.14)"

Discussion¶

Related Claims¶

• LLMs perform competently at comparing item difficulty but fall below random chance on comparing item discrimination — related
• Point-biserial item-total correlation is negatively related to test discriminating value and can be expressed via item and test difficulty and discrimination parameters — related
• Negative affective experience items were dropped due to floor effects and poor discrimination — related
• GLAT scores predicted performance on GenAI-supported tasks better than perceived ChatGPT proficiency — related
• Item analysis showed high self-reported engagement and subscale differences in discriminatory power, with social engagement items weakest and cognitive engagement items strongest — related