Classical test theory decision consistency estimation methods¶
Research Method · All research methods
Evidence · 5 claims (4 for, 1 against) · 1 study (1 design),q2· 0 of 1 report an effect size · 5 claims rest on one study
Description¶
The article organizes single-administration decision consistency estimation around three CTT methods: Livingston and Lewis (LL-DC), which assumes a beta-binomial distribution and uses effective test length; Peng-Subkoviak (PS-DC), a normal approximation extending Huynh's procedure; and Wolkowitz (W-DC), which uses 95% confidence intervals about observed scores. The article states "All three methods require total scores, reliability, and the cut score as input," with LL-DC additionally requiring minimum and maximum possible scores. Complexity of understanding ranges from high (LL-DC) to low (W-DC).
Accounts¶
- Three classical test theory decision consistency estimation methods differing in assumptions and complexity: The article organizes single-administration decision consistency estimation around three CTT methods: Livingston and Lewis (LL-DC), which assumes a beta-binomial distribution and uses effective test length; Peng-Subkoviak (PS-DC), a normal approximation extending Huynh's procedure; and Wolkowitz (W-DC), which uses 95% confidence intervals about observed scores. The article states "All three methods require total scores, reliability, and the cut score as input," with LL-DC additionally requiring minimum and maximum possible scores. Complexity of understanding ranges from high (LL-DC) to low (W-DC). (Wolkowitz et al. (2024))
Claims¶
- The LL-DC, PS-DC, and W-DC methods produce similar decision consistency estimates (within 0.044) for normally or close to normally distributed datasets [+M]
- For skewed datasets, W-DC and LL-DC produce similar and preferred estimates, while PS-DC diverges at high-frequency scores [+M]
- Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making it unsuitable for pass/fail decision consistency [-W]
- The greatest differences among the three DC estimates occur when the cut score is set at the mode of a skewed score distribution [+M]
- DC estimates tend to be lowest when the cut score is set at the peak of the score distribution, where inconsistent decisions are most likely [+W]
Related Research Methods¶
-
Key Sources¶
- Wolkowitz, A. A., & Smith, R. (2024). A Practical Comparison of Decision Consistency Estimates. Practical Assessment, Research & Evaluation, 29(6). https://pareonline.net/getvn.asp?v=29&n=6
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
classical-test-theory-decision-consistency-estimation-methods |
title |
Classical test theory decision consistency estimation methods |
description |
The article organizes single-administration decision consistency estimation around three CTT methods: Livingston and Lewis (LL-DC), which assumes a beta-binomial distribution and uses effective test length; Peng-Subkoviak (PS-DC), a normal approximation extending Huynh's procedure; and Wolkowitz (W- |
status |
review |
moved_from |
theory/ctt-decision-consistency-methods-taxonomy |
generated |
by process:sweep-kinds on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — wolkowitz-2024 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.