Model accuracy on Bloom classification drops with question text length, with LLMs degrading least¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Accuracy falls as text length grows: ML and BERT drop to 50% at 15-20 words on Scaria, while LLMs stay above 70% beyond 20 words. → Michael Lawrence Castanares 2026
Evidence¶
Michael Lawrence Castanares 2026¶
Michael Lawrence Castanares, Princess Ventures, and Allan Tan. (2026). Evaluation of Pre-trained Models for Pedagogical Assessment of NovelAI-Assisted Educational Questions. Preprint. https://arxiv.org/abs/2609.27749 doi:10.48550/arxiv.2609.27749
q2 · i? · design · r2
Accuracy-by-text-length analysis (Figure 3) across IID and OOD datasets, reported in Results Section 5.3; the authors characterize this as Bloom signal dilution. Only descriptive accuracies are printed.
"For OOD-Scaria, performance of ML and BERT models drop to 50% with text length 15-20 words. LLMs still exhibit dilution however their accuracy remained high (>70%) even for longer text (text length> 20 words)."
Discussion¶
Related Claims¶
- Supervised ML/DL Bloom question classifiers trained on one dataset drop substantially in weighted F1 when tested on unseen datasets — the general claim this finding bears on, with the studies behind it
- AI-generated OOD question datasets are longer, syntactically deeper, and overlap less with Bloom trigger verbs than human-curated IID data — related
- LLMs show robust zero- and few-shot Bloom classification of OOD AI-assisted questions compared with ML and BERT models — related
- In the random forest, word count was the most important readability feature while traditional readability formulas ranked near the bottom — related
- Feature-based ML Bloom classifiers trained on IID data degrade sharply on out-of-distribution AI-assisted questions — related
- Text splicing into single sentences improves ML and BERT Bloom classification on the Scaria OOD dataset — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
bloom-signal-dilution-with-text-length |
title |
Model accuracy on Bloom classification drops with question text length, with LLMs degrading least |
description |
Model accuracy on Bloom classification drops with question text length, with LLMs degrading least |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — michael-lawrence-castanares-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.