In the random forest, word count was the most important readability feature while traditional readability formulas ranked near the bottom¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? After 'Lesson', word count had the highest variable importance (28.99), and traditional readability metrics were not important, with SMOG the highest of these but below most other features. → Kole Norberg 2025 q2 i? Custom LSA magnitude ranked above magnitudes from spaCy or Word2Vec and above distance-between-texts measures. → Kole Norberg 2025 (2)

Evidence¶

Kole Norberg 2025¶

Kole Norberg, Husni Almoubayyed, and Stephen Fancsali. (2025). Linguistic Features Predicting Math Word Problem Readability Among Less-Skilled Readers. Proceedings of the 18th International Conference on Educational Data Mining, Palermo, Italy. https://doi.org/10.5281/zenodo.15870274

q2 · i? · associational · r2

Permutation-based variable importance from the random forest (Table 2): Word Count 28.99, the top readability feature after Lesson (100.00); Flesch Reading Ease 2.18 and LIX 0.00 rank lowest. Importance scores are printed, but no standardized effect size.

"First, traditional readability metrics were not important to the model's performance. The smog index was the highest ranking of these but fell below most other features."

Kole Norberg 2025 (2)¶

Kole Norberg, Husni Almoubayyed, and Stephen Fancsali. (2025). Linguistic Features Predicting Math Word Problem Readability Among Less-Skilled Readers. Proceedings of the 18th International Conference on Educational Data Mining, Palermo, Italy. https://doi.org/10.5281/zenodo.15870274

q2 · i? · associational · r2

Variable importance comparison within the same random forest: Custom Magnitude 20.13 versus Word2Vec Magnitude 6.05 and spaCy Magnitude 3.99 in Table 2. The authors read this as highlighting specialized MWP-corpus embeddings.

"Second, custom LSA magnitude emerged as more important to the model than magnitude calculated using SpaCy or Word2Vec and as more important than measures of distance between texts."

Discussion¶

Related Claims¶

• 4,446 of 9,421 MATHia word problems showed larger-than-expected error-rate gaps between less- and more-skilled readers and were flagged for potential readability concerns — related
• Partial dependence plots showed higher word count, more dependent clauses, and higher custom magnitude raised predicted flag probability, with non-linear relationships — related
• Academic word list count, word count, and Flesch-Kincaid grade level are the most important features for predicting cognitive engagement in discussion posts — related
• A random forest classified flagged versus non-flagged word problems with the best accuracy of five tested models (AUC = 0.75) — related
• Flagged word problems were shorter (lower word count) but had more sentences than non-flagged problems in descriptive statistics — related
• Rewriting MATHia word problems for struggling readers, by human experts or LLMs, sped completion by 30% and improved mastery rate — related
• Random Forest is the most commonly used NLP-supporting algorithm in LA, while deep learning models (LSTM, BERT) have been widely adopted since 2021 — related
• The three fine-tuned LLMs differ significantly in response length, readability, and similarity to posts — related
• Model accuracy on Bloom classification drops with question text length, with LLMs degrading least — related
• Product measures distinguish LLM-formulated text from classroom-formulated text regardless of how it was entered — related
• SHAP analysis identifies Hapax Ratio as the most influential stylometric feature in Random Forest predictions, followed by NSR, Noun Ratio, Adverb Ratio, and TTR — related
• Document length alone carries some signal (word-count classifier ROC-AUC 0.68), but the fixed-word sliding-window design largely prevents stylometric features from capturing length differences — related
• Engineered practice-balance features outperform raw activity counts: the Intensity Ratio ranks third among Codeforces predictors, and upsolving habit dominates the survey model — related
• Pairwise statistical tests confirm random forest's performance differences from other models are statistically significant (p < 0.05) — related
• PPB maintained comparable readability on three of four measures while increasing Flesch-Kincaid toward grade level — related
• Feature-based model predictive power derives from a large and varied set of features rather than a few dominant variables — related
• LLM-extracted cognitive and linguistic features add predictive value beyond readily available item metadata (grade, domain, item type, word count) — related
• Type-token ratio showed a non-monotonic relationship with predicted difficulty, with both very low and very high lexical diversity increasing predicted text difficulty
• Revising texts to improve readability-formula scores often produces nonexistent or insignificant comprehension gains — related