The Central Limit Theorem explains the normal-shape distortion: when reliability of site-specific estimates is low (r < .50), normal sampling error dominates the shape of the distribution of estimates¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q1
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? Because each site-specific estimate is normally distributed around its true value, when the reliability of estimates is relatively low (e.g., r < .50) the shape of the distribution of estimates is dominated by normal errors; only when reliability is high (e.g., r > .90) does the true distribution begin to dominate. → Henry May 2014

Evidence¶

Henry May 2014¶

Henry May. (2014). Distortions in Distributions of Impact Estimates in Multi-Site Trials: The Central Limit Theorem is Not Your Friend. SREE Fall 2014 Conference. https://eric.ed.gov/?id=ED562702

q1 · i? · theoretical · r3

Analytical argument grounded in the Central Limit Theorem: each site-specific estimate is distributed normally around its true estimate with variance equal to the squared standard error of the mean difference. The article reports Reading Recovery Empirical Bayes reliability of approximately .40, and simulation reliabilities of .38 (nj = 10) and .92 (nj = 200).

"When the reliability of the estimates is relatively low (e.g., r <.50), the shape of the distribution of estimates is dominated by the normal errors. Only when the reliability is high (e.g., r >.90) does the distribution of true site-specific impacts begin to dominate the shape of the distribution of impact estimates."

Discussion¶

Related Claims¶

• Random effects HLM models produce an unbiased estimate of the variance of true site-specific impacts regardless of the shape of the true distribution — related