Authentic intellectual work scoring rubrics with double-scoring design for inter-scorer reliability¶

Element · All elements
Evidence · 4 claims (2 for, 1 mixed, 1 against) · 4 studies (3 design, 1 associational), q2 · 0 of 4 report an effect size · 4 claims rest on one study

Description¶

The project scored assignments and student work using measurement rubrics created by writing and mathematics experts, one rubric set per subject (Documents 15 and 16). Each standard was scored 1 to 3 or 1 to 4, and each document received one score per standard. To control for rater differences, "half of the data documents needed to be double scored" so that "scorer severity" could be estimated, and matrix guidelines specified that all work be scored at least once per standard and that each possible pair of scorers work together at least once per standard.

Design Implications¶

Context¶

Requirements¶

• Teacher-scorers must be trained to reliability, with packets of about 9-11 documents double-scored so scorer severity can be statistically controlled

Constraints¶

• Scores were reliable enough for analysis only after Many-Facet Rasch adjustment, and cross-year comparison required equating because scorer severity drifted between years

Target Learners¶

• Grade 3, 6, 8, 9, and 10 students in Chicago public elementary and high schools

Target Learning Goals¶

• Measuring the authentic intellectual demand of assignments and the authenticity of student work

Affordances¶

• Authentic Intellectual Work Standards Framework

Claims¶

• Scorer severity and standard difficulty were statistically adjusted using Many-Facet Rasch analysis before assignment and work measures were compared [+W]
• In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable [-W]
• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions [+W]
• Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. [~W]

Related Elements¶

• Assignment-scoring standards and rubrics for authentic intellectual work in writing and mathematics
• Chicago Annenberg classroom assignment collection and scoring system

Examples¶

• Purposive school sampling plus careful document coding, cleaning, and archiving for longitudinal assignment and student-work research

Key Sources¶

• Wenzel, S., Nagaoka, J., Morris, L., Billings, S., & Fendt, C. (2002). Documentation of the 1996-2002 Chicago Annenberg Research Project Strand on Authentic Intellectual Demand Exhibited in Assignments and Student Work: A Technical Process Manual. Consortium on Chicago School Research. https://consortium.uchicago.edu/publications/documentation-1996-2002-chicago-annenberg-research-project-authentic-intellectual