Human annotators judge later-round CPIG items as improved in complexity but not in difficulty, with modest rater agreement¶

Claim · All claims
Evidence · 1 study (2 entries) · design r? · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? For complexity, round five shows a much larger fraction of items at the ideal level than round one; trends for difficulty are more static, with quite similar distributions. → Antonio Laverghetta Jr. 2024 q2 i? Intraclass correlations were 0.52 for complexity and 0.49 for difficulty, indicating modest rater agreement. → Antonio Laverghetta Jr. 2024 (2)

Evidence¶

Antonio Laverghetta Jr. 2024¶

Antonio Laverghetta Jr., Simone Luchini, Averie Linell, Roni Reiter-Palmon and Roger Beaty. (2024). The creative psychometric item generator: a framework for item generation and validation using large language models. CREAI 2024: Workshop on Artificial Intelligence and Creativity. https://arxiv.org/abs/2409.00202 doi:10.48550/arxiv.2409.00202

q2 · i? · design · r?

Human content review in which five annotators with prior creativity-rating experience rated CPIG items on five-point complexity and difficulty scales, blinded to round, using a missing-data approach (Results §4.4, Fig. 6). Findings are descriptive; no effect size is printed.

"For complexity, we see a definite improvement from round five, with a much larger fraction ofitems achieving the ideal complexity level than was present in round one. Trends are more staticfor difficulty as the distributions are quite similar to each other, especially at the ideal difficultylevel."

Antonio Laverghetta Jr. 2024 (2)¶

Antonio Laverghetta Jr., Simone Luchini, Averie Linell, Roni Reiter-Palmon and Roger Beaty. (2024). The creative psychometric item generator: a framework for item generation and validation using large language models. CREAI 2024: Workshop on Artificial Intelligence and Creativity. https://arxiv.org/abs/2409.00202

q2 · i? · design · r?

Reliability analysis of the same five-annotator content review (Results §4.4). The article prints ICCs of 0.52 (complexity) and 0.49 (difficulty) for absolute agreement on average ratings and attributes the modest agreement to the subjectivity of rating creativity.

"We obtained intraclass correlations of0.52 for complexity and0.49 for difficulty, for absoluteagreement on the average ratings, indicating a modest rater agreement."

Discussion¶

Related Claims¶

• LLM difficulty predictions for fraction items are less stable across repeated runs than for the overall item set — related
• CPIG's iterative exemplar selection consistently improves the originality of LLM responses to generated CPS items by the last round of item generation — related