Propensity score analysis estimated a Texas generalizability index of 0.91, and post-stratification yielded a population average treatment effect of 1.47 versus a conventional estimate of 1.44¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The experimental-plus-pilot sample matched the Texas school population well (β = 0.91), and post-stratification adjusted the average treatment effect only slightly upward. → Roschelle 2018 q2 i? 92% of Texas non-charter middle schools with 7th-grade mathematics have a good match in the study data. → Roschelle 2018 (2)

Evidence¶

Roschelle 2018¶

Roschelle, J., Tipton, E., Shechtman, N., Vahey, P. (2018). Generalizability of a Technology-Based Intervention to Enhance Conceptual Understanding in Mathematics (SimCalc Technical Report 10). Menlo Park, CA: SRI International. https://digitalpromise.dspacedirect.org/items/7a123d3c-22ec-46db-b7cf-709c3aa65583

q2 · i? · causal · r2

Propensity score analysis comparing 92 study schools with 1,713 Texas non-charter schools on 26 school-level covariates; Table 2 shows the post-stratification estimate (1.47, SE 0.15) versus the conventional estimate (1.44, SE 0.14).

"Using the data in this study plus the pilot, we estimate the generalizability index to be β = 0.91."

Roschelle 2018 (2)¶

Roschelle, J., Tipton, E., Shechtman, N., Vahey, P. (2018). Generalizability of a Technology-Based Intervention to Enhance Conceptual Understanding in Mathematics (SimCalc Technical Report 10). Menlo Park, CA: SRI International. https://digitalpromise.dspacedirect.org/items/7a123d3c-22ec-46db-b7cf-709c3aa65583

q2 · i? · causal · r2

Analysis of propensity score logit distributions (Figure 3) shows the remaining 8% of schools, in the long tail, have no similar schools in the study data.

"we can see that 92% of T exas non-charter middle schools with a 7th grade mathematics class have a good match in the study."

Discussion¶

Related Claims¶

• Propensity score stratification using multilevel models reduces selection bias from confounding variables and improves accuracy of charter school effect estimations — related
• Using baseline data strongly predictive of outcome measures considerably reduces bias in nonexperimental estimates of charter school achievement effects — related
• SimCalc treatment effects were not sensitive to school socioeconomic status or percentage of Hispanic students, and 91% of 7th-grade treatment classrooms exceeded the control-group mean gain — related
• An adapted SimCalc intervention in Florida produced strong learning gains (ES = 1.09), significantly higher posttests than predicted from the Texas control group — related
• An adapted SimCalc linear-function unit in England produced significant posttest gains versus Texas-control predictions (ES = 0.57), with no significant gain differences by poverty or prior achievement — related
• SimCalc replacement units produced statistically significant pretest-to-posttest gains in conceptual understanding with student-level effect sizes of .63, .50, and .56 in three Texas experiments — related