A mean-classification cut-score method outperformed a contrasting-groups-style mean method because it was more robust to outliers¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2002 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2005: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on an ERIC document (a report, conference paper or thesis). It is kept because 3 design pages cite it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? Method two, based on mean classification per portfolio, performed better than the category-mean method because it was more robust to outliers. → Olson 2002

Evidence¶

Olson 2002¶

Olson, B., Mead, R., & Payne, D. (2002). A report of a standard setting method for alternate assessments for students with significant disabilities (Synthesis Report 47). https://eric.ed.gov/?id=ED472306

q2 · i? · design · r2

Two cut-score computation methods were applied to panelist ratings of 15 portfolios per grade/content area. The report states "method two seemed to perform better in this application because it was more robust to outliers," while method one was heavily influenced by deviant ratings.

"Overall, method two seemed to perform better in this application because it was more robust to outliers."

Discussion¶

Related Claims¶

• Criterion-referenced tests require empirically-based construct validation studies — the general claim this finding bears on, with the studies behind it
• Performance classifications varied considerably across grades in both literacy and mathematics for the alternate portfolio assessment — related
• A body of work standard setting process produced classification agreement rates ranging from 71% to 97% for an alternate portfolio assessment — related
• Presenting ordered portfolios led panelists to shift toward a bookmarking-like mental process, contaminating independent classification — related
• Person-oriented and dimensional methods each predicted reading achievement significantly better than the categorical method — related
• Percent meeting standards is an imprecise metric whose cut-score concentration produced the Era 2 misconception that Chicago taught basic skills better than high-level skills — related