AI agents recover a smaller share of reference patterns from the synthetic WindowDash dataset than from the real-world Olist dataset¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across 600 simulations at six agent levels, all six agent levels recovered a smaller share of the nine reference patterns in WindowDash than in Olist when partial matches received half credit. → Bang An 2026

Evidence¶

Bang An 2026¶

Bang An, Maria Hamdani, Joseph Fox. (2026). DataCanvas-EDU: An Agentic Framework for Instructor-Guided Synthetic Data Generation in Business Analytics Education. https://arxiv.org/abs/2609.19617 doi:10.48550/arxiv.2609.19617

q2 · i? · design · r2

Simulation study comparing AI-agent pattern discovery on Olist and WindowDash: 50 simulations at each of six agent levels per dataset, totaling 600 simulations, each scored against nine dataset-specific reference patterns with half credit for partial matches. The authors note the "consistent discovery gap" favoring recovery on Olist.

"Figure 1 reveals a consistent discovery gap: with partial matches receiving half credit, all six agent levels recovered a smaller share of the reference patterns in WindowDash than in Olist."

Discussion¶

Related Claims¶

• The Olist and WindowDash simulation inputs are matched in scale and analytical scope, with overlapping distributions but differing duration-rating associations — related
• Designed patterns can rest on very small supporting subgroups: only three orders meet the Price-Rating Sensitivity conditions, with a mean rating of 2.0 — related
• The ASTRA benchmark yields measurable condition-differentiated patterns consistent with its simulation assumptions — related