LLM-simulated student ideas moderately overlap with teacher- and student-generated lesson ideas (mean cosine similarity 0.52), with larger LLMs slightly outperforming Mistral-7B¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across 49 lesson plans, the mean cosine similarity between LLM-generated and lesson ideas was 0.52 (SD = 0.07), and all six LLMs scored higher than Mistral-7B (M = 0.47). → Nguyen 2026 q2 i? Semantic similarity was significantly lower for high school lessons (M = 0.50) than for elementary (M = 0.54) and middle school (M = 0.55) lessons. → Nguyen 2026 (2)

Evidence¶

Nguyen 2026¶

Nguyen, H., & Cao, J. (2026). Exploring the Capacity of Large Language Models to Simulate Students' Scientific Thinking: Insights for Responsive Teaching. Journal of Science Education and Technology. https://doi.org/10.1007/s10956-026-10333-5

q2 · i? · design · r2

Computational evaluation of 8820 LLM-generated ideas clustered via embeddings and compared to lesson ideas from 49 OpenSciEd lesson plans. The mean cosine similarity was "0.52 (SD = 0.07)"; a likelihood ratio test (χ²(5) = 218.69, p <.001) showed all LLMs beat Mistral-7B (M = 0.47).

"Across 49 lesson plans spanning science domains and grade levels, the mean cosine similarity reflecting the overlap between LLM-generated and lesson ideas was 0.52 (SD = 0.07)."

Nguyen 2026 (2)¶

Nguyen, H., & Cao, J. (2026). Exploring the Capacity of Large Language Models to Simulate Students' Scientific Thinking: Insights for Responsive Teaching. Journal of Science Education and Technology. https://doi.org/10.1007/s10956-026-10333-5

q2 · i? · design · r2

Linear mixed models with activity ID as random intercept compared similarity across grade levels. High school similarity (M = 0.50) was significantly lower than elementary (β = 0.05, p =.01) and middle school (β = 0.05, p =.01), suggesting "weaker performance in matching the lessons' learning objectives for higher grade levels."

"cosine similarity differed significantly by grade level (elementary: M = 0.54, SD = 0.07; middle school: M = 0.55, SD = 0.07; high school: M = 0.50, SD = 0.05; χ²(2) = 10.94, p =.004; Fig. 6)."

Discussion¶

Related Claims¶

• GPT-4o and GPT-5.2 achieve higher accuracy and more fully correct answers on university IVDA multiple-choice questions than the smaller Mistral-7B and DeepSeek-R1 models — related
• Only 69% of LLM-simulated ideas met the target grade's language level, with elementary grades faring worst and Llama-4 performing best — related
• Teachers judged LLM-simulated student ideas realistic and most useful when the ideas sparked sensemaking about student thinking and suggested scaffolds — related
• Interest tailoring was structurally present in AI-generated readings: logs averaged 52.24 tailoring markers per 10,000 words and showed moderate divergence (cosine similarity 0.50–0.61) — related
• Large language models can support learning and teaching across education levels, from reading and writing support for elementary students to professional training — related
• LLM-generated ideas diverge from lesson ideas in grounding, misconception substance, vocabulary, uncertainty markers, and analogy use — related