For skewed datasets, W-DC and LL-DC produce similar and preferred estimates, while PS-DC diverges at high-frequency scores¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? PS-DC estimates were closer to LL-DC for the simulated datasets, but W-DC estimates were closer to LL-DC for the operational datasets. → Wolkowitz 2024 q2 i? The W-DC and LL-DC methods are recommended for skewed data; PS-DC will likely overestimate DC when the cut score is set at a frequently earned score. → Wolkowitz 2024 (2)

Evidence¶

Wolkowitz 2024¶

Wolkowitz, A. A., & Smith, R. (2024). A Practical Comparison of Decision Consistency Estimates. Practical Assessment, Research & Evaluation, 29(6). https://pareonline.net/getvn.asp?v=29&n=6

q2 · i? · design · r2

Overall comparison across five datasets (two simulated, three operational). The article reports the "PS- DC estimates were closer to the LL -DC estimates for the simulated datasets" and W-DC closer for operational datasets, with closeness percentages of 65-71% and 61-74% of scores respectively.

"In comparing the W-DC and PS-DC methods to LL-DC, the PS- DC estimates were closer to the LL -DC estimates for the simulated datasets, but the W-DC estimates were closer to the LL- DC estimates for the operational datasets."

Wolkowitz 2024 (2)¶

Wolkowitz, A. A., & Smith, R. (2024). A Practical Comparison of Decision Consistency Estimates. Practical Assessment, Research & Evaluation, 29(6). https://pareonline.net/getvn.asp?v=29&n=6

q2 · i? · design · r2

Recommendation drawn from both the simulated and operational dataset results, summarized in Table 12. The article attributes PS-DC's divergence to its bivariate normality assumption, which is violated by skewed score distributions.

"the W- DC and LL -DC methods are recommended for skewed data. The PS -DC method may be use d for skewed data but will likely overestimate the DC value if the cut score is set at a score frequently earned by examinees."

Discussion¶

Related Claims¶

• The greatest differences among the three DC estimates occur when the cut score is set at the mode of a skewed score distribution — related
• DC estimates tend to be lowest when the cut score is set at the peak of the score distribution, where inconsistent decisions are most likely — related
• The LL-DC, PS-DC, and W-DC methods produce similar decision consistency estimates (within 0.044) for normally or close to normally distributed datasets — related