Direct zero-shot GPT-4o difficulty ratings beat a grade-level dummy regressor and correlate strongly with true Rasch difficulties overall, for K-5 math and reading items¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2 · i3 large
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i3 On the holdout sample, direct LLM estimates showed lower error than the dummy regressor and correlated strongly with true difficulties for math (r = .83) and reading (r = .81). → Pooya Razavi 2025 q2 i3 For reading, direct LLM estimates produced lower RMSE and MAE than the dummy regressor benchmarks. → Pooya Razavi 2025 (2)

Evidence¶

Pooya Razavi 2025¶

Pooya Razavi, Sonya Powers. (2025). Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms. Edmentum. https://doi.org/10.1016/j.ijaied.2026.100015

q2 · i3 · design · r2

Holdout-sample evaluation of the direct LLM estimation approach for math (rescaled GPT-4o ratings entered in per-grade regressions). Errors beat the dummy regressor, and the article reports "a strong correlation between the true and predicted difficulty estimates, when collapsing all grades together (r = .83)".

"both error indices (RMSE = 0.91, MAE = 0.72) were lower than the benchmarks (1.01 and 0.81, respectively), indicating that the GPT estimates perform better than dummy regressor models."

Pooya Razavi 2025 (2)¶

Pooya Razavi, Sonya Powers. (2025). Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms. Edmentum. https://doi.org/10.1016/j.ijaied.2026.100015

q2 · i3 · design · r2

Reading holdout results for the direct approach: "The overall RMSE (0.86) and MAE (0.69)" beat the dummy benchmarks, and predicted and true estimates were highly correlated (r = .81; Figure 1).

"The overall RMSE (0.86) and MAE (0.69) were considerably lower than the benchmarks (1.04 and 0.84, respectively), indicating that using the difficulty estimates generated by the LLM provides better accuracy compared to estimates generated by a dummy regressor."

Discussion¶

Related Claims¶

• Direct Llm Accuracy Varies By Grade Early Grades Worse
• Feature Based Llm Models Outperform Direct Estimation And Benchmarks
• LLM-extracted cognitive and linguistic features add predictive value beyond readily available item metadata (grade, domain, item type, word count) — related
• 91% of LLM-simulated ideas fell within grade-level knowledge scope, but within-scope proportions were significantly lower for elementary and middle school than high school — related
• Epi2Diff achieves the best performance across all four item-difficulty test sets and all reported metrics, outperforming SLM fine-tuning, LLM prompting, and supervised LLM adaptation baselines — related
• Hypothesis-based predictors match or beat black-box difficulty regressors on unseen questions — related
• Zero-shot LLMs perform only marginally above random in five-class competency classification but exceed 70% accuracy on binary classification — related
• Models with similar rank correlations differ substantially in estimating difficulty magnitude (RMSE and MAE) — related
• Perceived task difficulty ratings show a strong relation to Rasch-estimated relative task difficulty, supporting construct validity of the single-item scale — related