LLM-generated assessment conversations may reveal more information than intended and present scoring challenges compared with pre-structured conversations¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The authors identify a downside of LLM conversations: they may provide more information than desired during assessment, and less pre-structured conversation may challenge scoring. → Diego Zapata-Rivera 2024

Evidence¶

Diego Zapata-Rivera 2024¶

Diego Zapata-Rivera, Carol Forsyth, Edith Aurora Graf and Yang Jiang. (2024). Designing and Evaluating Evidence-Centered Design based Conversations for Assessment with LLMs. Leveraging Large Language Models for Next Generation Educational Technologies, CEUR Workshop Proceedings. https://ceur-ws.org

q2 · i? · design · r2

Authors' observation from the case study comparing LLM and RegExp/LSA conversation fragments: unlike tutoring dialogues, CBAs aim to elicit rather than give away answers, yet LLM turns such as the soda-can analogy explanation may reveal content the assessment intends the student to supply.

"However, there is a downside in that currently the LLM produced conversations may be providing more information than desired during the conversation-based assessment. Furthermore, the ability to continue conversations in less pre-structured ways may present challenges for scoring."

Discussion¶

Related Claims¶

• In a case study, LLM-generated assessment conversations handled common student speech acts more naturalistically than RegExp/LSA conversations — related
• LLM capabilities enabled family education systems across conversational, embodied, and spatial interface forms to generate support from the context of an unfolding interaction — a broader claim this one bears on