Scorer severity and standard difficulty were statistically adjusted using Many-Facet Rasch analysis before assignment and work measures were compared¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2002 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Assignment and student-work measures were developed with Many-Facet Rasch analysis to adjust ratings for scorer severity, standard difficulty, and step difficulties between score categories. → Wenzel 2002

Evidence¶

Wenzel 2002¶

Wenzel, S., Nagaoka, J., Morris, L., Billings, S., & Fendt, C. (2002). Documentation of the 1996-2002 Chicago Annenberg Research Project Strand on Authentic Intellectual Demand Exhibited in Assignments and Student Work: A Technical Process Manual. Consortium on Chicago School Research. https://consortium.uchicago.edu/publications/documentation-1996-2002-chicago-annenberg-research-project-authentic-intellectual

q2 · i? · associational · r2

In the measure-development stage, the project used the Many-Facet Rasch model, noting that besides adjusting for rater severity and standard difficulty, "another advantage of the Rasch model is that it allows for missing data." Final assignment measures exist in a logit metric with standard errors, converted to a 0-10 scale for reporting.

"the student work and teacher assignment measures were developed using Many-Facet Rasch Analysis with statistical software specifically designed for this model, FACETS."

Discussion¶

Related Claims¶

• Raters differed significantly in severity/leniency in the MFRM analysis — related
• CTT and MFRM yield similar item difficulty results for open-ended questions regardless of crossed or nested design — related
• CTT and MFRM yield a positive, perfect correlation between item difficulty indices for open-ended items under a crossed design — related
• Items ranked from easiest to most difficult are identical under CTT and MFRM — related
• Scorers found all rubric dimensions important, with feedback bearing mainly on the scoring and training process rather than rubric design — related
• Scorers had difficulty distinguishing rubric scoring from everyday grading and AP scoring, and training took longer than expected — related