Use SR and OB measures together as diagnostic tools in AI literacy professional development, before and after training¶

Strategy · All strategies
Evidence · 2 claims (2 for) · 2 studies (1 quant-synthesis, 1 associational), q2 · 0 of 2 report an effect size · 2 claims rest on one study

Description¶

Because SR and OB measures are weakly correlated and cannot be used interchangeably, the authors recommend embedding both validated instruments into AI literacy PD programs to monitor growth and detect calibration issues. The instruments "serve as diagnostic tools that can be embedded into AI literacy PD programs, both before and after training" to assess changes in perceived and demonstrated competencies. Insights can inform differentiated professional development or targeted scaffolding for subgroups of teachers showing overestimation, underestimation, or alignment.

Design Implications¶

Context¶

Requirements¶

• Both SR and OB instruments administered within the same shared AI literacy framework
• Sufficient sample size for latent profile analysis to identify subgroups

Constraints¶

• The analysis did not separate teachers by pre-service or in-service status or teaching experience
• The objective assessment was not tailored to specific subjects or grade levels

Target Learners¶

• K–12 pre-service and in-service teachers in AI literacy professional development

Target Learning Goals¶

• Accurate self-assessment and growth of AI literacy competencies across Concept, Use, Evaluate, and Ethics dimensions

Claims¶

• Adding a cross-factor composite from a teacher study yields r = .079, still with an interval including zero, and latent profiles reveal over- and underestimation of competence [+W]
• Latent profile analysis of combined SR and OB factors identifies six distinct teacher profiles spanning overestimation, underestimation, and alignment [+W]

Related Strategies¶

• Use a mixed-methods assessment sequence: diagnostic inventory, structured verification tasks, and discipline-specific critique or workflow-design assignments
• Use external instruments rather than teacher judgment to measure learning strategies for differentiation decisions

Examples¶

-

Key Sources¶

• Shan Zhang, Ruiwei Xiao, Anthony F. Botelho, Guanze Liao, Thomas K. F. Chiu, John Stamper, and Kenneth R. Koedinger. 2026. How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026). ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785088