GenAI novices develop prompting strategies through trial and error, starting from zero-shot prompts and refining them after unsatisfactory outputs¶

Claim · All claims
Evidence · 1 study · qualitative r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Student pairs in a text linguistics assignment formulated an initial zero-shot prompt and iteratively refined it after judging LLM outputs unsatisfactory. → Nicola Brocca 2026

Evidence¶

Nicola Brocca 2026¶

Nicola Brocca, Davide Garassino. (2026). LLMs in text linguistics teaching: an exploratory study with genAI novices in higher education. Computers and Education Open 11 (2026) 100414. https://doi.org/10.1016/j.caeo.2026.100414

q2 · i? · qualitative · r2

Deductive coding of Dataset 1, five pair-produced reports from a home assignment using GPT-3.5, coded against categories derived from prompting-strategy literature and the HITL workflow. The analysis found "a trial-and-error approach" with initial single zero-shot prompts followed by iterative refinement.

"The prompting strategies employed by the five groups were developed through a trial-and-error approach. Initially, students formulated a single prompt closely aligned with the wording of the assignment"

Discussion¶

Related Claims¶

• When refined prompts failed, novices overgeneralized and concluded LLMs could not detect anaphors, attributing failure to the model rather than to their prompting — related
• Zero-shot LLM errors follow four recurring patterns: over-interpretation, failure to detect relevant information, hallucination, and failure to generate a response — related
• Chain-of-thought prompting yields higher-quality GenAI feedback on argumentative essays than Zero-shot prompting and teacher feedback — related
• Zero-shot GenAI feedback quality is statistically comparable to teacher feedback quality — related
• Participants' prompting was often limited, with poorly formulated prompts leading to vague or unhelpful outputs — related
• Some participants' ChatGPT interactions followed a pattern of inquiry-based learning with iterative prompt refinement — a narrower finding that bears on this claim
• Working with imperfect AI outputs under mentor supervision was followed by students developing skepticism and independent debugging skills — related
• First-cycle underperformance was attributed to low-quality student-generated questions that failed to represent journal article patterns — related