Distributions of estimated intervention effects overstate the spread of true effects; adjusted empirical Bayes estimates reduce the spread by about 16% for cumulative semesters enrolled¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? For cumulative semesters enrolled through two semesters, the standard deviation of original OLS impact estimates (0.057) is 16% larger than the standard deviation of adjusted empirical Bayes estimates (0.049), which equals the estimated true-effect standard deviation. → Michael J. Weiss 2023

Evidence¶

Michael J. Weiss 2023¶

Michael J. Weiss, Marie-Andrée Somers, & Colin Hill. (2023). Empirical Benchmarks for Planning and Interpreting Causal Effects of Community College Interventions. Journal of Postsecondary Student Success 3 (1). https://doi.org/10.33009/fsop_jpss132759

q2 · i? · quant-synthesis · r2

Methodological comparison in the Results section using Figure 1, which plots original OLS, empirical Bayes, and adjusted empirical Bayes impact estimates for 39 interventions on cumulative semesters enrolled. The article reports the OLS spread "is 16% larger" than the adjusted empirical Bayes spread (0.057 vs 0.049).

"First, notice that the spread of the ˆOrig jB is 16% larger than the spread of ˆAEB jB . Specifi- cally, /g11/g12/g32 0.057ˆOrig jSD B , whereas the /g11/g12/g32 0.049ˆAEB jSD B ."

Discussion¶

Related Claims¶

• Cross-site standard deviations of ITT effects were estimated for education and workforce interventions using data from 16 large multisite randomized controlled trials — a broader claim this one bears on
• The Detroit Promise Path effect of 1.8 credits earned after two semesters sits at the 83rd percentile of the distribution of community college intervention effects, indicating a quite large effect — related
• For credits earned in semester 1, the estimated distribution of true intervention effects has a mean of 0.48 credits, a cross-intervention standard deviation of 0.55 credits, and a 75th percentile of 0.60 credits — related
• Cross-intervention variation in community college intervention effects is substantial: for first-semester credits earned, the true-effect standard deviation (0.55 credits) exceeds the mean effect (0.48 credits) — related
• Random effects HLM models produce an unbiased estimate of the variance of true site-specific impacts regardless of the shape of the true distribution — related