The objective-based AI literacy assessment demonstrates psychometric stability with high reliability (KR-20 = 0.862) and approximate unidimensionality¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? A 1PL Rasch analysis of the 25-item objective-based AI literacy assessment with 288 teachers showed high internal consistency (KR-20 = 0.862), acceptable item fit after removing five misfitting items, and approximate unidimensionality. → rasch-1

Evidence¶

rasch-1¶

Shan Zhang, Ruiwei Xiao, Anthony F. Botelho, Guanze Liao, Thomas K. F. Chiu, John Stamper, and Kenneth R. Koedinger. 2026. How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026). ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785088

q2 · i? · associational · r2

Rasch (1PL) analysis of the objective-based assessment (N = 288) found high internal consistency, with "Internal consistency was high (KR-20 = 0.862)". Item difficulties ranged from −1.83 to 1.1 logits; five misfitting items were removed; PCA of residuals supported approximate unidimensionality (first contrast eigenvalue 0.44).

"Internal consistency was high (KR-20 = 0.862), which indicates reliable measurement. Item-person targeting, as illustrated in the Wright Map (see Figure 1), showed that item difficulties were well aligned with teacher ability levels"

Discussion¶

Related Claims¶

• The objective-based measure yields a confirmed three-factor structure with acceptable fit in the full sample — related
• Teachers' self-reported AI literacy correlates only weakly with their objectively measured performance (r = 0.07–0.24) — related
• Exploratory factor analysis of survey items yields a unidimensional 9-item AI pedagogical orientation scale with strong reliability — related
• Post-2024 instrument development expanded objective AI-literacy measurement, which had been dominated by self-report scales — a broader claim this one bears on
• Latent profile analysis of combined SR and OB factors identifies six distinct teacher profiles spanning overestimation, underestimation, and alignment — related
• A large-scale long-term study examined the measurement stability of Rasch measurement scales that had been in place for almost 40 years — related
• Maze-task reading performance from 1,258 first and second graders yields a reliable Rasch text-complexity scale — related