Replacing natural books with synthetic textbooks improves the 28-benchmark mean by +1.09, with gains spanning all four categories¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Full improves 19 of 28 benchmarks over Natural Books with a +1.09 overall mean, improving all four category means. → Jiawen Tao 2026 q2 i? Bootstrap intervals remain positive for all three aggregate comparisons, and one-sided sign tests give p=0.044, 0.0019, and 0.000014 for Full against Natural Books, Split, and Rephrase. → Jiawen Tao 2026 (2)

Evidence¶

Jiawen Tao 2026¶

Jiawen Tao, Miao Peng, Yaoming Li, Xiaokun Yuan, Mengzhou Wu, Wenhan Yu, Guoan Wang, Nuo Chen, Tong Yang, Maxm Pan. (2026). Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training. arXiv preprint. https://arxiv.org/abs/2607.28109 doi:10.48550/arxiv.2607.28109

q2 · i? · causal · r2

MoE mid-training study comparing the synthetic-book corpus against the original book slice under the same 16B-token allocation. The article reports the printed category deltas and 19/28 benchmark wins.

"Full improves all four category means over Natural Books—STEM ( +1.86), reasoning (+1.49), code ( +1.07), and knowledge ( +0.43)—and 19/28 benchmarks ( +1.09 overall)."

Jiawen Tao 2026 (2)¶

Jiawen Tao, Miao Peng, Yaoming Li, Xiaokun Yuan, Mengzhou Wu, Wenhan Yu, Guoan Wang, Nuo Chen, Tong Yang, Maxm Pan. (2026). Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training. arXiv preprint. https://arxiv.org/abs/2607.28109

q2 · i? · causal · r2

Robustness check on the same results: benchmark-level bootstrap intervals remain positive for all three aggregate comparisons; the article notes this does not estimate training-run variance.

"One-sided sign tests give p=0.044, 0.0019, and 0.000014 for Full against Natural Books, Split, and Rephrase, respectively."

Discussion¶

Related Claims¶

• The book-organization benefit transfers to Llama3-8B, where Full outperforms RandomConcat (+0.86) and Natural Books (+1.51) — related
• Preserving book-level document structure during mid-training improves downstream performance beyond identical content split into sections (+1.02 mean) — related
• Structured book synthesis outperforms matched independent document rephrasing (+1.17 mean), while rephrasing alone ties natural books — related
• Pipeline components play complementary roles: hierarchical generation has the largest effect, grounding improves specificity and factual accuracy, and cluster-informed TOC planning improves structure and audience fit — related