91% of LLM-simulated ideas fell within grade-level knowledge scope, but within-scope proportions were significantly lower for elementary and middle school than high school¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Overall, 91% of simulated ideas were within the corresponding grade-level knowledge scope, with GPT-4o and Llama-4 performing best and GPT-5-mini most likely to exceed scope. → Nguyen 2026 q2 i? Compared to high school, within-scope proportions were significantly lower for middle school (OR = 0.14) and elementary school (OR = 0.07). → Nguyen 2026 (2)

Evidence¶

Nguyen 2026¶

Nguyen, H., & Cao, J. (2026). Exploring the Capacity of Large Language Models to Simulate Students' Scientific Thinking: Insights for Responsive Teaching. Journal of Science Education and Technology. https://doi.org/10.1007/s10956-026-10333-5

q2 · i? · design · r2

LLM-as-a-judge coding (GPT-5, κ = 0.69 with a human coder; robustness check with Claude-Sonnet-4, κ = 0.64) of knowledge scope against NGSS grade-band expectations. GPT-5-mini ideas were more likely to exceed scope (Δ proportion = −0.27 vs GPT-4o and Llama-4).

"Overall, 91% of the simulated ideas were within the corresponding knowledge scope (Fig. 10). Specifically, GPT-4o and Llama-4 performed the best: their simulated ideas fell within the expected knowledge scope for the target grades."

Nguyen 2026 (2)¶

Nguyen, H., & Cao, J. (2026). Exploring the Capacity of Large Language Models to Simulate Students' Scientific Thinking: Insights for Responsive Teaching. Journal of Science Education and Technology. https://doi.org/10.1007/s10956-026-10333-5

q2 · i? · design · r2

Generalized linear mixed models with grade as fixed effect (χ² = 17.18, p <.001). Within each LLM, significant grade differences appeared only for GPT-5-mini (χ² = 9.18, p =.01), e.g., overly complex atom-level ideas in a fourth-grade energy lesson.

"Compared to high school, the within-scope proportion was significantly lower for middle school (Δ proportion = −0.09; OR = 0.14, 95% CI [0.03, 0.68], p =.02) and elementary school (Δ proportion = −0.16; OR = 0.07, 95% CI [0.02, 0.33], p <.001)."

Discussion¶

Related Claims¶

• Only 69% of LLM-simulated ideas met the target grade's language level, with elementary grades faring worst and Llama-4 performing best — related
• Iterative prompting to revise overly complex responses raised readability alignment, with most regenerated responses falling at or below target grade levels — related
• Teachers judged LLM-simulated student ideas realistic and most useful when the ideas sparked sensemaking about student thinking and suggested scaffolds — related
• Direct zero-shot GPT-4o difficulty ratings beat a grade-level dummy regressor and correlate strongly with true Rasch difficulties overall, for K-5 math and reading items — related