Teachers' self-reported AI literacy correlates only weakly with their objectively measured performance (r = 0.07–0.24)¶

Claim · All claims
Evidence · 1 study · associational r2 · q2 · i2 medium
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i2 Correlations between the three objective-based factors and the four self-report factors were consistently weak, ranging from r = 0.07 to r = 0.24, indicating perceptions and demonstrated competence are weakly associated. → corr-1

Evidence¶

corr-1¶

Shan Zhang, Ruiwei Xiao, Anthony F. Botelho, Guanze Liao, Thomas K. F. Chiu, John Stamper, and Kenneth R. Koedinger. 2026. How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026). ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785088

q2 · i2 · associational · r2

Correlational analysis in the full sample (N = 288) found SR–OB correlations "consistently weak, ranging from 𝑟= 0.07to 𝑟= 0.24"; the strongest was Concept_SR with Conceptual Understanding_OB (r = 0.24) and the weakest Evaluate_SR with Practical and Ethical Use_OB (r = 0.07).

"Correlations between the three OB factors—Conceptual Understanding_OB,Capability Evaluation_OB, andPractical and Ethical Use_OB—and the four SR factors were consistently weak, ranging from 𝑟= 0.07to 𝑟= 0.24"

Discussion¶

Related Claims¶

• The objective-based AI literacy assessment demonstrates psychometric stability with high reliability (KR-20 = 0.862) and approximate unidimensionality — related
• SR–OB profile patterns differ by prior AI literacy experience: overestimation appears only among experienced teachers and a low–low profile only among inexperienced teachers — related
• Latent profile analysis of combined SR and OB factors identifies six distinct teacher profiles spanning overestimation, underestimation, and alignment — a narrower finding that bears on this claim
• The objective-based measure yields a confirmed three-factor structure with acceptable fit in the full sample — related
• Adding a cross-factor composite from a teacher study yields r = .079, still with an interval including zero, and latent profiles reveal over- and underestimation of competence — a narrower finding that bears on this claim
• Post-2024 instrument development expanded objective AI-literacy measurement, which had been dominated by self-report scales — a broader claim this one bears on
• Perceived and objectively demonstrated AI literacy show only limited alignment in the same samples (pooled r = .055, interval including zero) — related
• Self-reported AI literacy was strongest in ethics and evaluation and weakest in self-efficacy, indicating uncertainty in applying AI to complex tasks — related
• Learning gains were uncorrelated with prior self-reported AI literacy and database literacy, suggesting AI integration benefited students irrespective of prior perceptions — related
• Students' self-reported AI literacy was moderately negatively correlated with objectively measured AI competency, suggesting potential overestimation of AI abilities — related
• Students' self-reported perception of peer assessment's benefit is virtually uncorrelated with their actual measured benefit — a narrower finding that bears on this claim
• Prior Knowledge Needed For Accurate Self Assessment — a broader claim this one bears on
• Institutional initiatives, demographics, teaching practices, and information sources show weak or null associations with AI pedagogical orientation and use — related
• Perceived usefulness of ChatGPT, an invalid cue, is associated with overestimation of performance — a narrower finding that bears on this claim
• Recent AI-in-education studies often rely on subjective or performance-during-support measures and weak comparison designs, limiting claims about learning effects — a narrower finding that bears on this claim
• Teachers' ratings of children's creativity correlate poorly with psychometric performance, and teachers tend to prefer appropriate answers over novel ones — related