Combining human and LLM-generated responses at a 1:1 ratio yields more accurate educational item calibration¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? An IRT-based study found that mixing human and synthetic responses at a 1:1 ratio produced more accurate item calibration than synthetic responses alone, though high-performing LLMs showed narrower zones of ZPD limiting reflection of human variability. → Luis Marquez-Carpintero 2025

Evidence¶

Luis Marquez-Carpintero 2025¶

Luis Marquez-Carpintero, Alberto Lopez-Sellers, Miguel Cazorla. (2025). Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI. https://arxiv.org/abs/2511.06078 doi:10.48550/arxiv.2511.06078

q2 · i? · review · r2

The review reports an Item Response Theory study by Liu et al. [2025] testing six LLMs (GPT-3.5, GPT-4, LLaMA 2, LLaMA 3, Gemini-Pro, Cohere Command R Plus) on 20 university-level algebra questions, comparing estimated model ability against real students. No effect size is printed.

"However, combining human and synthetic responses at a 1:1 ratio resulted in more accurate item calibration, suggesting that LLMs can serve as an effective complement in educational assessment."

Discussion¶

Related Claims¶

• Both proposed adjustment methods (IEA and BA) reduce RMSE of item difficulty estimates from direct 2PL-IRT fitting, with BA yielding more consistent (lower-variance) estimates in simulation — related
• Ensemble modeling of multiple LLM scorers yields performance gains in automated divergent thinking scoring — related
• Combining the pattern detector with an LLM fallback does not improve joint detection accuracy over the LLM alone — related