Retrained BERT relies on high-impact nouns, verbs, and adjectives for correct predictions, but this reliance is muted on the AF dataset¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Chi-square tests show dataset domain is associated with high-impact verbs (χ2 = 31.44, p<0.001) and nouns (χ2 = 8.54, p<0.014) but not adjectives (p = 0.085). → Michael Lawrence Castanares 2026

Evidence¶

Michael Lawrence Castanares 2026¶

Michael Lawrence Castanares, Princess Ventures, and Allan Tan. (2026). Evaluation of Pre-trained Models for Pedagogical Assessment of NovelAI-Assisted Educational Questions. Preprint. https://arxiv.org/abs/2609.27749 doi:10.48550/arxiv.2609.27749

q2 · i? · design · r2

LIME interpretability analysis of the retrained BERT model across the three datasets (Annex 3, Figure 5); correct predictions on Lau and Scaria relied on tokens with LIME weights > 0.50, while AF showed muted impact. Only test statistics printed.

"Using Chi-square test confirmed high association between dataset domain and high-impact verbs (χ2 = 31.44,p<0.001 ) and Nouns (χ2 = 8.54,p<0.014 ) but not on adjectives (p= 0.085 )."

Discussion¶

Related Claims¶

• Model retraining on labeled OOD data provides the largest improvement in Bloom classification across models and datasets — related
• Appending learning objectives to questions improves Bloom classification on the AF dataset — related
• AI-generated OOD question datasets are longer, syntactically deeper, and overlap less with Bloom trigger verbs than human-curated IID data — related
• Dual-Lens attention aligns with expert pedagogical focus, allocating most weight to Bloom's taxonomy verbs and problem-solving steps — related
• Replacing context-specific terms with syntactic descriptors and entity categories allows NLP rules for argumentation-component segmentation that are less context dependent — related
• Feature-based ML Bloom classifiers trained on IID data degrade sharply on out-of-distribution AI-assisted questions — related
• LLMs show robust zero- and few-shot Bloom classification of OOD AI-assisted questions compared with ML and BERT models — related