Supervised ML/DL Bloom question classifiers trained on one dataset drop substantially in weighted F1 when tested on unseen datasets¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? TFPOSIDF+SVM cross-dataset weighted F1 drops by an average of 0.25 relative to within-dataset test scores. → Abdolali Faraji 2026 q2 i? Fine-tuned BERT cross-dataset weighted F1 drops by an average of 0.28 relative to within-dataset test scores. → Abdolali Faraji 2026 (2)

Evidence¶

Abdolali Faraji 2026¶

Abdolali Faraji, Mohammadreza Molavi, Zohreh Rasoulkhani, Mohammadreza Tavakoli, and Gábor Kismihók. (2026). Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs. https://arxiv.org/abs/2606.13684 doi:10.48550/arxiv.2606.13684

q2 · i? · causal · r2

Cross-dataset generalization evaluation (RQ1) over five Bloom-annotated datasets, each serving once as training source with an 80/20 split. The SVM model's unseen-dataset weighted F1 scores ranged 0.40–0.70, an "average of 0.25" drop.

"For TFPOSIDF+SVM, test scores range from 0.68 to 0.82 depending on the training dataset, but performance decreases on unseen datasets by an average of 0.25, ranging between 0.40 and 0.70."

Abdolali Faraji 2026 (2)¶

Abdolali Faraji, Mohammadreza Molavi, Zohreh Rasoulkhani, Mohammadreza Tavakoli, and Gábor Kismihók. (2026). Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs. https://arxiv.org/abs/2606.13684

q2 · i? · design · r2

Same cross-dataset evaluation protocol (RQ1); the fine-tuned BERT reference model's weighted F1 fell from 0.76–0.91 within-dataset to 0.31–0.76 on unseen datasets, an average drop of 0.28.

"Similarly, fine-tuned BERT achieves test scores between 0.76 and 0.91, which drop on average by 0.28 when evaluated on other datasets, ranging from 0.31 to 0.76."

Studies bearing on this claim¶

Can a model tag questions by Bloom level reliably enough to use the tags in a course? This table gathers the studies behind this claim and the 7 narrower claims that bear on it: 3 distinct studies (2 causal, 1 design), q2, 0 of 3 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• LLMs are more stable than supervised models across dataset shifts, though they do not surpass within-dataset fine-tuned performance — related
• Transfer between two specific datasets is asymmetric: Sangodiah-trained models transfer unusually well to Gani questions — a narrower finding that bears on this claim
• Feature-based ML Bloom classifiers trained on IID data degrade sharply on out-of-distribution AI-assisted questions — related
• The framework is robust to 15% ASR transcription noise and retains 99.6% of its ICS under zero-shot cross-domain transfer to EdNet-Math — related
• Offloading is decodable from metacognitive monitoring and control brain processes — related
• Grading skill transfers across exams: an adapter fine-tuned on one exam improves grading on the unseen other, though transfer stops short of in-domain performance — related