Published knowledge test Alphas show distributional bias with underrepresentation of low Alphas¶

Claim · All claims
Evidence · 1 study (3 entries) · quant-synthesis r2–r3 · q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q3 i? The distribution of transformed Alphas is right-skewed (skewness = 0.97, p < .001), indicating underrepresentation of low Alphas in published studies. → Peter A. Edelsbrunner 2025 q3 i? There are fewer short tests with low Alphas than the Spearman–Brown formula predicts. → Peter A. Edelsbrunner 2025 (2) q3 i? There are increased frequencies of Alphas just above the cut-offs of 0.70, 0.80, and 0.90. → Peter A. Edelsbrunner 2025 (3)

Evidence¶

Peter A. Edelsbrunner 2025¶

Peter A. Edelsbrunner, Bianca A. Simonsmeier, Michael Schneider. (2025). The Cronbach’s Alpha of Domain‑Specific Knowledge Tests Before and After Learning: A Meta‑Analysis of Published Studies. Educational Psychology Review. https://doi.org/10.1007/s10648-024-09982-y

q3 · i? · quant-synthesis · r2

After Bonett transformation, the distribution of Alphas showed "moderate right-skewness" with skewness index 0.97 (p < .001). After removing nine exceptionally high Alphas from two studies, skewness remained moderate at 0.44 (p = .003).

"the distribution indicates moderate right-skewness, as indicated by a skewness index of 0.97 (SE = 0.14; bootstrapped p < 0.001)"

Peter A. Edelsbrunner 2025 (2)¶

Peter A. Edelsbrunner, Bianca A. Simonsmeier, Michael Schneider. (2025). The Cronbach’s Alpha of Domain‑Specific Knowledge Tests Before and After Learning: A Meta‑Analysis of Published Studies. Educational Psychology Review. https://doi.org/10.1007/s10648-024-09982-y

q3 · i? · quant-synthesis · r3

Comparison of the empirical Alpha-by-test-length curve with the Spearman–Brown predicted curve showed the empirical curve deviates for tests below 20 items, indicating "an underrepresentation of short tests with low Alphas."

"this finding indicates an underrepresentation of short tests with low Alphas compared to what can be expected for an unbiased distribution"

Peter A. Edelsbrunner 2025 (3)¶

Peter A. Edelsbrunner, Bianca A. Simonsmeier, Michael Schneider. (2025). The Cronbach’s Alpha of Domain‑Specific Knowledge Tests Before and After Learning: A Meta‑Analysis of Published Studies. Educational Psychology Review. https://doi.org/10.1007/s10648-024-09982-y

q3 · i? · quant-synthesis · r2

Frequency distributions showed "moderate yet salient peaks in the numbers of reported Alphas just above" the cut-offs of 0.70, 0.80, and 0.90, consistent with researchers modifying tests to exceed cut-offs.

"Figs. 2 and 4 show moderate yet salient peaks in the numbers of reported Alphas just above these three cut-offs"

Discussion¶

Related Claims¶

• Alpha of knowledge tests is substantially related to the number of items in the test — related
• Meta-analytic average Cronbach's Alpha of domain-specific knowledge tests is 0.85 with expected item intercorrelation of 0.22 — related
• Fixed alpha cutoff values create publication bias so knowledge tests with low alphas are underrepresented in the literature — possibly the same claim (merge candidate)