Random effects HLM models produce an unbiased estimate of the variance of true site-specific impacts regardless of the shape of the true distribution¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2017: tool and model accuracy changes quickly

Subclaims¶

q2 i? Monte Carlo results suggest random effects HLM models do a reasonable job of producing an unbiased estimate of the variance of true impacts regardless of the shape of the true distribution of impact estimates. → Henry May 2014

Evidence¶

Henry May 2014¶

Henry May. (2014). Distortions in Distributions of Impact Estimates in Multi-Site Trials: The Central Limit Theorem is Not Your Friend. SREE Fall 2014 Conference. https://eric.ed.gov/?id=ED562702

q2 · i? · theoretical · r3

Monte Carlo simulation finding: although the shape of the distribution could not be recovered, the article reports random effects HLM "do[es] a reasonable job of producing an unbiased estimate of treatment effect variation" across uniform, lognormal, and negative lognormal true distributions.

"Results from the Monte Carlo simulations suggests that random effects HLM models do a reasonable job of producing an unbiased estimate of treatment effect variation (i.e., the variance of the true impacts ) regardless of the shape of the true distribution of impact estimates."

Discussion¶

Related Claims¶

• The Central Limit Theorem explains the normal-shape distortion: when reliability of site-specific estimates is low (r < .50), normal sampling error dominates the shape of the distribution of estimates — related
• Cross-site standard deviations of ITT effects were estimated for education and workforce interventions using data from 16 large multisite randomized controlled trials — related
• Distributions of estimated intervention effects overstate the spread of true effects; adjusted empirical Bayes estimates reduce the spread by about 16% for cumulative semesters enrolled — related
• The Reading Recovery i3 multi-site RCT finds large positive overall effects on reading outcomes with substantial variation in impacts across sites — related