LLM-extracted cognitive and linguistic features add predictive value beyond readily available item metadata (grade, domain, item type, word count)¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Full models combining metadata with LLM-rated features improved on metadata-only models: reading RF cut RMSE 17.6% and math GBM cut RMSE 11.2% versus the strongest metadata-only baselines, on average. → Pooya Razavi 2026

Evidence¶

Pooya Razavi 2026¶

Razavi, P., & Powers, S. (2026). Estimating item difficulty using large language models and tree-based machine learning algorithms. International Journal of Artificial Intelligence in Education, 36. https://doi.org/10.1016/j.ijaied.2026.100015

q2 · i? · design · r2

Reading holdout comparison of full feature-based models against metadata-only models trained with identical algorithms and procedures. For math, GBM reduced RMSE "relative to the strongest benchmark (i.e., the metadata-only gradient boosting model) by 11.2% on average".

"Compared to the dummy regressor, the random forest reduced RMSE by 31.6% on average, and improved upon the metadata-only random forest by 17.6% on average."

Discussion¶

Related Claims¶

• Feature Based Llm Models Outperform Direct Estimation And Benchmarks
• Direct zero-shot GPT-4o difficulty ratings beat a grade-level dummy regressor and correlate strongly with true Rasch difficulties overall, for K-5 math and reading items — related
• Feature-based model predictive power derives from a large and varied set of features rather than a few dominant variables — related
• Combining semantic and process features yields the best difficulty prediction; semantic embeddings alone trail the full model by 7.4–12.1 macro-F1 points on SAT Math — related
• Episode-composition and transition features contribute predictive process evidence beyond semantics, with Epi.+Tran. reaching 0.136 permutation importance versus 0.084 for Sem. in the Both (Role) setting — related
• In the random forest, word count was the most important readability feature while traditional readability formulas ranked near the bottom — related
• Test item construction, including avoidance of cognate L1 equivalents, often affects KVL difficulty scores beyond the genuine difficulty of the words — related
• All three assertion sources (word, corpus-derived, construct-derived) contribute to predictive accuracy — a narrower finding that bears on this claim
• The explainable feature-based model with LLM-extracted features outperforms the closed-track encoder baseline by a wide margin — related