Lower-performing LLMs show higher variance across pedagogy subject categories and peak in Technology and General categories¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? The bottom 15 performing models demonstrate much higher performance variance across subjects and consistently achieve their best scores in the Technology and General categories, a specialization absent in the top 15 models. → Lelièvre 2025

Evidence¶

Lelièvre 2025¶

Lelièvre, M., Waldock, A., Liu, M., Valdes Aspillaga, N., Mackintosh, A., Ogando Portela, M. J., Lee, J., Atherton, P., Ince, R. A. A., & Garrod, O. G. B. (2025). Benchmarking the Pedagogical Knowledge of Large Language Models. https://arxiv.org/abs/2506.18710 doi:10.48550/arxiv.2506.18710

q2 · i? · design · r2

Subject-level sub-score analysis of CDPK accuracy for the top 15 and bottom 15 performing models, shown in Figure 10. The article reports that "the bottom 15 models demonstrate a much higher performance variance across subjects" and peak in Technology and General categories, while top models show less variance and more diverse peak subjects.

"Firstly, the bottom 15 models demonstrate a much higher performance variance across subjects, evidenced by their wider 'Performance Range' bars. Secondly, these lower-performing models consistently achieve their best scores (indicated by stars) in the Technology and General categories."

Discussion¶

Related Claims¶

• Temperature-construct interactions: constructs with moderate theoretical coherence benefited from higher temperatures, while well-defined constructs required deterministic settings — related
• SEND and CDPK benchmark results correlate highly (r = 0.94), but higher-performing models do relatively better on general pedagogy than SEND — related
• The two education-specialized models lead neither education module, and the improvement from education-specific post-training is small relative to differences in general capability — related
• Model performance varies by task category, with Basic Geometry easiest and Spiral consistently the most challenging category — related