A body of work standard setting process produced classification agreement rates ranging from 71% to 97% for an alternate portfolio assessment¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2002 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2005: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on an ERIC document (a report, conference paper or thesis). It is kept because 3 design pages cite it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? Panelist classification agreement rates for the alternate portfolio standard setting ranged from 71% to 97% across grades and subjects. → Olson 2002

Evidence¶

Olson 2002¶

Olson, B., Mead, R., & Payne, D. (2002). A report of a standard setting method for alternate assessments for students with significant disabilities (Synthesis Report 47). https://eric.ed.gov/?id=ED472306

q2 · i? · design · r2

Single-state standard setting study with five or six raters per panel and fifteen portfolios per grade/content area, yielding 75 or 90 ratings per panel. The report states "The classification agreement rates ranged from 71% to 97%" and that the source of variation could not be identified from one study.

"The classification agreement rates ranged from 71% to 97%. With a single study, it is not possible to identify the source of the variation."

Discussion¶

Related Claims¶

• The authors assert that innovative assessment measures capture POG competencies more fully than traditional standardized tests — the general claim this finding bears on, with the studies behind it

• Criterion-referenced tests require empirically-based construct validation studies — the general claim this finding bears on, with the studies behind it

• Performance classifications varied considerably across grades in both literacy and mathematics for the alternate portfolio assessment — related
• Presenting ordered portfolios led panelists to shift toward a bookmarking-like mental process, contaminating independent classification — related
• A mean-classification cut-score method outperformed a contrasting-groups-style mean method because it was more robust to outliers — related
• MAP Reading Fluency SWCPM scores classify students into ARM proficiency categories with exact match rates of 74.6–90.8%, kappa 0.60–0.85, and overall accuracy 0.81–0.96 — related