Higher-quality prompt engineering skills predict the quality of LLM output in goal-directed use of generative AI tools¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Students with higher-quality prompt engineering skills produce higher-quality LLM output, indicating prompt engineering is a required skill for goal-directed use of generative AI. → Knoth 2024

Evidence¶

Knoth 2024¶

Knoth, N., Tolzin, A., Janson, A., & Leimeister, J. M. (2024). AI literacy and its implications for prompt engineering strategies. Computers and Education: Artificial Intelligence. https://doi.org/10.1016/j.caeai.2024.100225

q2 · i? · associational · r?

Study of non-expert students interacting with LLM-based AI systems through prompt engineering; the article reports that "higher-quality prompt engineering skills predict the quality of LLM output". No effect size or test statistic is printed in the available text.

"The results show that higher-quality prompt engineering skills predict the quality of LLM output, suggesting that prompt engineering is indeed a required skill for the goal-directed use of generative AI tools."

Discussion¶

Related Claims¶

• Certain aspects of AI literacy play a role in higher-quality prompt engineering by non-expert students — related
• Effective AI use in nursing education depends on prompting skills, which participants identify as critical for obtaining accurate and reliable outputs — possibly the same claim (merge candidate)
• Reaching the qualitative SOLO phase in GenAI interaction required repeated meta-level learner interventions, demanding the very self-regulatory skills the tool was expected to support — related
• LLM responses that are incomplete or incorrect prompt students to question, verify and improve the information, fostering critical thinking through cycles of validation and correction — related
• Within prompt–response AI interfaces, students without guidance rarely move beyond prompt tuning, yielding interactions of limited educational value and increased reliance on outputs — related
• Non-expert students exhibit intuitive behaviors towards LLM-based AI systems when prompt engineering — related
• More complex prompts tend to elicit superior results with advanced LLM versions (authors' observation) — related
• The authors observe that more specific physical descriptions in generation prompts tend to produce more physically coherent AI-generated motion — a narrower finding that bears on this claim
• Students in a scaffolded LLM-collaboration course improved significantly on initial argument quality, prompt engineering, and response to AI feedback from the first to the fifth assignment — related