Feature-based model predictive power derives from a large and varied set of features rather than a few dominant variables¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Post-hoc importance analyses (permutation importance for random forests; SHAP for GBMs) showed predictions draw on both objective metadata and granular LLM-rated features. → Pooya Razavi 2026

Evidence¶

Pooya Razavi 2026¶

Razavi, P., & Powers, S. (2026). Estimating item difficulty using large language models and tree-based machine learning algorithms. International Journal of Artificial Intelligence in Education, 36, 100015. doi:10.1016/j.ijaied.2026.100015

q2 · i? · design · r2

Post-hoc feature importance analysis (Table 8, top 20 features per subject) using permutation-based importance for random forests and SHAP for gradient boosting machines. The authors caution that "feature importance is model-specific".

"the models' predictive power is not derived from a few dominant variables, but rather from a large and varied set of features that includes both objective item metadata (e.g., Grade Level, Word Count) and the more granular LLM-rated features (e.g., Use of Visuals, Vocabulary Complexity)."

Discussion¶

Related Claims¶

• Feature Based Llm Models Outperform Direct Estimation And Benchmarks
• Random Forest and SHAP analyses identify Gamma and Beta band power at temporal and frontal channels (T3, P3, F7, T4, T4, F8, F7) as the most critical features for EEG familiarity classification — related
• SHAP analysis identifies Hapax Ratio as the most influential stylometric feature in Random Forest predictions, followed by NSR, Noun Ratio, Adverb Ratio, and TTR — related
• Academic word list count, word count, and Flesch-Kincaid grade level are the most important features for predicting cognitive engagement in discussion posts — related
• In the random forest, word count was the most important readability feature while traditional readability formulas ranked near the bottom — related
• Engineered practice-balance features outperform raw activity counts: the Intensity Ratio ranks third among Codeforces predictors, and upsolving habit dominates the survey model — related
• LLM-extracted cognitive and linguistic features add predictive value beyond readily available item metadata (grade, domain, item type, word count) — related
• For activity-defined engagement prediction in DTA logs, the activity-only Baseline remained strongest, and permutation importance was dominated by activity indicators — reports the opposite
• All parts of the item were important for predicting response time — a narrower finding that bears on this claim
• Discipline-general academic vocabulary (AWL use) supports cognitive engagement identification and may aid generalization across courses — related
• SHAP analysis of the explainable model identifies production frequency and spelling difficulty as high-impact predictors of KVL difficulty, with spelling difficulty varying strongly by L1 — related
• Episode-composition and transition features contribute predictive process evidence beyond semantics, with Epi.+Tran. reaching 0.136 permutation importance versus 0.084 for Sem. in the Both (Role) setting — related