When refined prompts failed, novices overgeneralized and concluded LLMs could not detect anaphors, attributing failure to the model rather than to their prompting¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Students discontinued prompt refinement after concluding the LLMs were unable to detect anaphors, an overgeneralization the authors note contradicts prior research showing LLMs outperform human novices on this task. → Nicola Brocca 2026 q2 i? Only one of eight students attributed unsatisfactory results to weaknesses in their own prompt strategy. → Nicola Brocca 2026 (2)

Evidence¶

Nicola Brocca 2026¶

Nicola Brocca, Davide Garassino. (2026). LLMs in text linguistics teaching: an exploratory study with genAI novices in higher education. Computers and Education Open 11 (2026) 100414. https://doi.org/10.1016/j.caeo.2026.100414

q2 · i? · design · r2

Authors' discussion of Dataset 1 (five pair reports) interprets the discontinuation of refinement as overgeneralization, noting previous research indicates LLMs generally outperform human novices on anaphora detection, so the students' conclusion underestimated model capabilities.

"When the revised prompts did not yield better results, the students concluded that the LLMs were unable to detect anaphors and discontinued further prompt refinement. This conclusion appears to reflect an overgeneralization from their own prompting experience."

Nicola Brocca 2026 (2)¶

Nicola Brocca, Davide Garassino. (2026). LLMs in text linguistics teaching: an exploratory study with genAI novices in higher education. Computers and Education Open 11 (2026) 100414. https://doi.org/10.1016/j.caeo.2026.100414

q2 · i? · qualitative · r2

Qualitative content analysis of the eight end-of-term reflections (Dataset 2, approximately 2800 words) found that in output evaluation, "only one student attributed the lack of satisfactory results to weaknesses in their prompt strategy".

"Interestingly, only one student attributed the lack of satisfactory results to weaknesses in their prompt strategy"

Discussion¶

Related Claims¶

• GenAI novices develop prompting strategies through trial and error, starting from zero-shot prompts and refining them after unsatisfactory outputs — related
• When prompts failed, students most commonly reported clarifying intent or specifying more detail, and reflecting on the problem depiction, rather than tracing code or examining test cases — related
• Participants' prompting was often limited, with poorly formulated prompts leading to vague or unhelpful outputs — related
• Self-explanation promotes transfer of general patterns but can reduce recall of details and cause overgeneralization — related
• The report attributes stagnant learning results to program implementation rather than to the ed-tech tools themselves — related