Text-based reflective use from Copilot Critical Engagement responses does not predict productive engagement, with limited coder reliability and a TA ceiling effect (exploratory null)¶

Claim · All claims
Evidence · 2 studies · 2 associational r1–r2 · q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? Technology acceptance did not significantly predict RUItext (b = 0.690, p = 0.118), and RUItext did not significantly predict productive engagement (b = 0.149, p = 0.202) or project score; the text-based indirect effect was not significant. → Wang S and Zhao J 2026 q1 i? Initial intercoder agreement was limited (exact agreement 57.6%, linear-weighted κ = 0.421, ordinal α = 0.495), and the Copilot TA measure showed a ceiling effect (28.9% maximum scores; 81.6% at least 4.5). → Wang S and Zhao J 2026 (2)

Evidence¶

Wang S and Zhao J 2026¶

Wang S and Zhao J (2026) Reflective AI use and student engagement in AI-supported programming: technology acceptance, programming self-efficacy, and behavioral trace evidence. Front. Psychol. 17:1929887. https://doi.org/10.3389/fpsyg.2026.1929887

q1 · i? · associational · r1

Exploratory complete-case regressions on the Copilot field-study dataset (66 nonblank Critical Engagement responses of 114 records), double-coded by two blinded human coders into a four-level RUItext index. The article reports "Technology acceptance did not significantly predictRUItext (b = 0.690, SE = 0.435, and p = 0.118)"; the indirect estimate was 0.101 with a percentile interval from –0.022 to 0.304.

"Technology acceptance did not significantly predictRUItext (b = 0.690, SE = 0.435, and p = 0.118), and RUItext did not significantly predict productive engagement or project score."

Wang S and Zhao J 2026 (2)¶

Wang S and Zhao J (2026) Reflective AI use and student engagement in AI-supported programming: technology acceptance, programming self-efficacy, and behavioral trace evidence. Front. Psychol. 17:1929887. https://doi.org/10.3389/fpsyg.2026.1929887

q1 · i? · associational · r2

Reliability of the two coders' independent ratings before adjudication of the 66 nonblank responses; the article states these estimates "indicate limited initial agreement" and do not justify treating RUItext as a fully validated scale. A separate ceiling-effect quote reports 33 of 114 students (28.9%) at the maximum TA score and 93 (81.6%) at least 4.5.

"Exact agreement was 57.6%, linear-weighted Cohen’sκ was 0.421, and Krippendorff’s ordinal α was 0.495."

Discussion¶

Related Claims¶

• Reflective AI use predicts programming self-efficacy and engagement, while PSE does not predict engagement (survey SEM) — reports the opposite
• Technology acceptance directly predicts reflective AI use and learning engagement in AI-supported programming (survey SEM, N=131) — related
• Hypothesized indirect effects from technology acceptance to engagement through reflective use, PSE, and the serial RU–PSE path are not significant (null result) — related
• Students engaged deeply with LLM feedback, with coded evidence of reflection, rebuttal, and acceptance in the large majority of responses — related
• Engagement in deep learning activities, as measured by NSSE's DEEP Learning scale, shows no statistically significant relationship with objectively measured critical thinking at the end of the first year — related