Disciplinary AI interaction competencies are observable in student response artifacts and scorable with rubrics grounded in student data and expert knowledge¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? For two physics tasks deployed in a course, the authors developed rubrics and argue that student AI interaction competencies are observable in response artifacts and scorable; no scored student data are reported. → Gao 2026

Evidence¶

Gao 2026¶

Gao, Y., Chen, Z., Li, M., & Zhai, X. (2026). Generative AI as a Design Variable: An Evidence-Centered Framework for Principled Governance in STEM Assessment. https://arxiv.org/abs/2605.24837 doi:10.48550/arxiv.2605.24837

q2 · i? · design · r2

Design demonstration in an undergraduate introductory physics course: two Require-GenAI tasks were deployed, and four rubrics (D1 0–4, D2 0–3 per task) anchored in an expert answer key were developed to show that the tasks' artifacts can be scored. The paper reports no scores or analysis of the deployed students' responses, and no effect sizes or test statistics; the support is a worked design demonstration rather than an empirical comparison.

"we present two task designs deployed in an introductory physics course and demonstrate that disciplinary AI interaction competencies are observable in student response artifacts and can be scored using defensible rubrics grounded in student data and expert knowledge."

Discussion¶

Related Claims¶

• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — a narrower finding that bears on this claim
• Artifact-grounded agent responses are rated significantly higher than transcript-only responses on groundedness, analytical depth, helpfulness, relevance, and overall quality — related
• LLMs support the shift from memorization-focused to competency-based assessment by generating rubrics, varied assessment items and authentic scenario-based tasks — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related