In Study 2, inter-rater agreement for rating creative problem solving ideas was good to excellent across indicators¶

Claim · All claims
Evidence · 1 study · associational r2 · q3
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Intraclass correlation coefficients for the top 1 to top 3 ratings ranged from 0.75 to 0.87 for originality, 0.68 to 0.80 for completeness, and 0.79 to 0.89 for practicality. → Van Hooijdonk 2023

Evidence¶

Van Hooijdonk 2023¶

Van Hooijdonk, M., Mainhard, T., Kroesbergen, E. H., & Van Tartwijk, J. (2023). Creative problem solving in primary school students. Learning and Instruction, 88, 101823. https://doi.org/10.1016/j.learninstruc.2023.101823

q3 · i? · associational · r2

Psychometric analysis in Study 2, in which two postgraduate raters who received 16 hours of training rated students' self-selected top-three ideas on 5-point scales using a modified Consensual Assessment Technique. The ICCs for originality "ranged from 0.75 to 0.87, indicating excellent agreement", with comparable values for completeness and practicality. No effect size is printed.

"The ICCs for originality ranged from 0.75 to 0.87, indicating excellent agreement. The ICCs for completeness ranged from 0.68 to 0.80, indicating good to excellent agreement. The ICCs for practicality ranged from 0.79 to 0.89, again indicating excellent agreement."

Discussion¶

Related Claims¶

• Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• The LoU Interview shows high interrater reliability, and the rating procedure was streamlined over two years without loss of agreement — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• LLM observer ratings converge excellently with human expert judgments in applying the CAARS observer instrument to generated narratives — related