LLMs are more stable than supervised models across dataset shifts, though they do not surpass within-dataset fine-tuned performance¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? LLM performance shows substantially less sensitivity to dataset shifts than the supervised models evaluated in RQ1. → Abdolali Faraji 2026 q2 i? GPT-5 consistently outperforms GPT-5-mini, but the gap is modest (average weighted F1 only 0.03 lower for the mini model). → Abdolali Faraji 2026 (2)

Evidence¶

Abdolali Faraji 2026¶

Abdolali Faraji, Mohammadreza Molavi, Zohreh Rasoulkhani, Mohammadreza Tavakoli, and Gábor Kismihók. (2026). Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs. https://arxiv.org/abs/2606.13684 doi:10.48550/arxiv.2606.13684

q2 · i? · design · r2

RQ2 discussion comparing Table 3 LLM results with Table 2 supervised results: LLMs did not surpass within-dataset fine-tuned performance but were comparatively stable across diverse datasets.

"Compared to the supervised models evaluated in RQ1, LLM performance shows substantially less sensitivity to dataset shifts, indicating stronger cross-dataset robustness."

Abdolali Faraji 2026 (2)¶

Abdolali Faraji, Mohammadreza Molavi, Zohreh Rasoulkhani, Mohammadreza Tavakoli, and Gábor Kismihók. (2026). Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs. https://arxiv.org/abs/2606.13684

q2 · i? · causal · r2

RQ2 model-level comparison across all prompting strategies and datasets (Table 3): GPT-5 beat GPT-5-mini consistently, with the mini model averaging weighted F1 only 0.03 lower.

"GPT-5 consistently performs better than GPT-5-mini; however, the performance gap remains modest, making GPT-5-mini a viable alternative when computational efficiency or cost is a concern, with an average weighted F1-score only 0.03 lower."

Discussion¶

Related Claims¶

• EchoPrompt maintains cross-proxy robustness, averaging 87.06% AUROC across seven proxy models and outperforming DNA-DetectLLM by 2.70% AUROC on average — related
• Supervised ML/DL Bloom question classifiers trained on one dataset drop substantially in weighted F1 when tested on unseen datasets — related
• Grading skill transfers across exams: an adapter fine-tuned on one exam improves grading on the unseen other, though transfer stops short of in-domain performance — a narrower finding that bears on this claim
• LLM diagnostic performance drops sharply when student responses contain vague rather than evident evidence — related
• In one essay-feedback system, fine-tuned models produced no usable deep-level feedback output, so only directly prompted LLMs were evaluated; the authors suggest prompting may be more effective than supervised fine-tuning for this task — reports the opposite
• Transfer between two specific datasets is asymmetric: Sangodiah-trained models transfer unusually well to Gani questions — related