Faculty testing of AI chatbots on real student homework found feedback quality uneven, prompting-dependent, and focused on surface features¶

Claim · All claims
Evidence · 1 study · design r2 · q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q1 i? When faculty tested AI chatbots on anonymized student homework, AI solved every problem but its corrective feedback was uneven, depended heavily on prompting, and focused on surface features rather than deeper conceptual issues. → Perl-Nussbaum 2026

Evidence¶

Perl-Nussbaum 2026¶

Perl-Nussbaum, D., & Finkelstein, N. D. (2026). A Workshop Series for Effective Use of AI in Uncertain Times: Building a Physics Faculty Learning Community. https://arxiv.org/abs/2609.23887 doi:10.48550/arxiv.2609.23887

q1 · i? · design · r2

Small-group hands-on testing during Session 3, in which faculty tried prompts on real anonymized student homework from a modern physics course (technical and conceptual items). The article reports feedback was "uneven and depended heavily on prompting" and needed rubric guidance to be reliable.

"AI easily solved every problem, but its ability to correct student work and give useful feedback was uneven and depended heavily on prompting"

Discussion¶

Related Claims¶

• Usability testing with educators and developers identified chatbot strengths for independent challenge-solving but limited feedback for low-participation learners — related
• Empirical gains from AI tools in Arab EL2 classrooms are most consistent for surface-level outcomes, while higher-order writing quality and speaking proficiency show mixed results contingent on teacher mediation — a broader claim this one bears on
• Feedback prompted with ELL status narrows to surface-level grammar and mechanics corrections, assuming limited language competency rather than addressing argument and reasoning — a broader claim this one bears on
• A six-session faculty learning community on AI was implemented in a physics department, engaging nineteen faculty across biweekly meetings — related
• Students primarily benefit from AI chatbots in three key areas: homework and study assistance, personalized learning, and skills development — reports the opposite
• AI chatbots provide students detailed feedback, step-by-step homework solutions, and self-quizzing support — reports the opposite
• AI chatbots support skills development in writing, problem-solving, and group discussion facilitation — reports the opposite
• Teachers who focused on knowledge-building principles showed deeper understanding than those focused on surface procedures or technical aspects — related