Evaluate AI literacy interventions with behavioral measures of LLM interaction in addition to self-reports¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends that AI literacy interventions be evaluated with behavioral measures in addition to self-reports, because self-reports "may miss important differences in actual use". The recommended behaviors include prompt acceptance or reformulation, correctness judgments of LLM responses, and follow-up questioning, treated as indicators of applied, procedural AI literacy rather than declarative knowledge alone.

Design Implications¶

Context¶

Requirements¶

• Task settings in which students interact with an LLM so that prompt decisions, correctness judgments, and follow-up questions can be logged

Constraints¶

• The article frames prompt acceptance as an experimental probe, noting students would often write their own prompts from scratch in ecological classroom use

Target Learners¶

• middle-school students using LLM-based chatbots for school tasks

Target Learning Goals¶

• applied regulation of LLM interaction during problem solving

Related Strategies¶

• Evaluate AI standardized patient systems with both endpoint and process-sensitive outcomes
• Evaluate LLM-supported learning beyond binary access by measuring enacted interaction and preserving no-help assessments
• Combine cognitive tasks and ecological teacher questionnaires when measuring children's inhibition for academic research
• Use external instruments rather than teacher judgment to measure learning strategies for differentiation decisions

Examples¶

-

Key Sources¶

• Olivier Clerc, Rania Abdelghani, Chloé Desvaux, Eliott Poisson, Pierre-Yves Oudeyer, Hélène Sauzéon. (2026). Teaching Students to Question the Machine: Short-Term Effects of an AI Literacy Workshop on Middle-School Students’ Regulation of LLM Interaction. ALIT4ALL 2026: 2nd International Workshop on AI Literacy Education For All. https://osf.io/fyu7c/overview