A single configurable analytics codebase replicates prior research findings across three educational AI assistant domains¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? Configuration payloads constructed for the JW, VERA, and SAMI domains produced pipeline outputs that replicate each domain's previously reported data analysis. → Y. Bai 2025

Evidence¶

Y. Bai 2025¶

Y. Bai, P. Thajchayapong, A. Goel. (2025). Generalizing a Highly Configurable Analytics Pipeline to Replicate and Support Educational Research Across Multiple Domains. https://arxiv.org/abs/2605.30303 doi:10.48550/arxiv.2605.30303

q2 · i? · design · r2

Replication demonstration reported in the Results section: payloads for the Jill Watson, VERA, and SAMI domains were run through the pipeline, and "The outputs generated by the pipeline replicate the findings of each domain's data analysis." Result files were stored per AI tool for dashboard display.

"Configuration payloads were constructed for each research domain and passed as input to the Analytics Pipeline. The outputs generated by the pipeline replicate the findings of each domain's data analysis."

Discussion¶

Related Claims¶

• The pipeline's contingency table option applied identically across all three domains reproduced previous descriptive findings — a narrower finding that bears on this claim
• Statistical power analysis for Welch's t-test originally developed for the VERA domain was extended to the SAMI domain through configuration alone — a narrower finding that bears on this claim
• Lack of standardised AI reporting undermines reproducibility, peer review, meta-analysis, and impact — related
• Previously identified effects in single digit multiplication, including the problem size effect and the tie effect, replicate in an integrated item-difficulty analysis. — a narrower finding that bears on this claim
• AI-ALOE technology deployments were associated with improved adult learner outcomes, including higher course grades and retention for Jill Watson and higher-quality written work for SMART, though VERA outcomes varied by course structure — related