The greatest differences among the three DC estimates occur when the cut score is set at the mode of a skewed score distribution¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? In the skewed simulated dataset, the greatest difference in DC estimates (0.101) occurred at the mode (score 30), where LL-DC estimated 0.825, PS-DC 0.844, and W-DC 0.743. → Wolkowitz 2024 q2 i? In the most skewed operational dataset (Dataset B), the greatest difference (0.063) occurred at score 63, where LL-DC estimated 0.777, PS-DC 0.839, and W-DC 0.796. → Wolkowitz 2024 (2)

Evidence¶

Wolkowitz 2024¶

Wolkowitz, A. A., & Smith, R. (2024). A Practical Comparison of Decision Consistency Estimates. Practical Assessment, Research & Evaluation, 29(6). https://pareonline.net/getvn.asp?v=29&n=6

q2 · i? · design · r2

Numerical comparison for the negatively skewed simulated dataset (Sim 2, 1,000 examinees, 31 items, reliability 0.841). The greatest difference of 0.101 occurred at the mode; all other estimates for the skewed datasets were within 0.029 of each other.

"At this score of 30 out of 31, the LL -DC method estimated the DC index to be 0.825, while the PS-DC and W-DC methods estimated the index to be 0.844 and 0.743, respectively."

Wolkowitz 2024 (2)¶

Wolkowitz, A. A., & Smith, R. (2024). A Practical Comparison of Decision Consistency Estimates. Practical Assessment, Research & Evaluation, 29(6). https://pareonline.net/getvn.asp?v=29&n=6

q2 · i? · design · r2

Operational licensure dataset B (1,509 examinees, 80 items, reliability 0.786, skewness -0.85). The greatest difference among methods, 0.063, occurred at score 63 near the mode; cut scores 58-65 showed differences greater than 0.04.

"At this score, the LL-DC method estimated the DC index to be 0.777 while the PS-DC and W-DC methods estimated the index to be 0.839 and 0.796, respectively."

Discussion¶

Related Claims¶

• For skewed datasets, W-DC and LL-DC produce similar and preferred estimates, while PS-DC diverges at high-frequency scores — related
• The LL-DC, PS-DC, and W-DC methods produce similar decision consistency estimates (within 0.044) for normally or close to normally distributed datasets — related
• DC estimates tend to be lowest when the cut score is set at the peak of the score distribution, where inconsistent decisions are most likely — a broader claim this one bears on