An LLM-based question-specificity classifier reached .830 knowledge and .829 scaffold accuracy without fine-tuning¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Among three tested models, Gemini-3.5-Flash performed best consistently across the three programming tasks for both knowledge specificity (.830) and scaffold specificity (.829) against human-annotated data. → Hyoungwook Jin 2026

Evidence¶

Hyoungwook Jin 2026¶

Hyoungwook Jin, Weirui Peng, Jieun Han, Q. Vera Liao, and Xu Wang. (2026). HelpCoach: Scaffolding Targeted AI Help-Seeking During Problem-Solving. arXiv. https://arxiv.org/abs/2609.28918 doi:10.48550/arxiv.2609.28918

q2 · i? · causal · r2

Technical evaluation comparing three LLM classifiers against human-annotated data from Study 1 (739 questions) on binary knowledge- and scaffold-specificity classification. Gemini-3.5-Flash achieved the highest accuracy; the article notes it achieved reasonable accuracy without fine-tuning on a large dataset.

"Among the three models, Gemini-3.5-Flash performed the best consistently across the three programming tasks for both knowledge specificity (.830) and scaffold specificity (.829)."

Discussion¶

Related Claims¶

• FATE benchmarks rank Gemini 2.5 Flash highest among commercial AI tutors, with actionability weakest for all models — related
• LLMs show robust zero- and few-shot Bloom classification of OOD AI-assisted questions compared with ML and BERT models — related
• Gemini-3-flash achieved the highest rubric-item accuracy, ranging from 89% on Course 2 to over 99% on Course 1 — related
• Gemini 2.0 Flash validated as a reliable LLM annotator of feedback focus types — related
• One student group compared outputs across three LLMs and judged Gemini more accurate overall, though it still failed fine-grained distinctions — related