The adapted 4B model outperforms zero-shot cloud baselines on lexical and semantic measures despite lower raw accuracy¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Cloud baselines (GPT-4o-mini 69.5%, Gemini-2.5-FL 82.9% accuracy) score lower Token F1 and BERTScore than the adapted 4B model, which reaches 50.9% accuracy. → Rashid Azraf Jahin 2026

Evidence¶

Rashid Azraf Jahin 2026¶

Rashid Azraf Jahin, Saadman Sajid, Khan Raiyan Ibne Reza, Sumaiya Tabassum Nimi. (2026). PhysicsMate: A Curriculum-Grounded Bengali Benchmark for Secondary Physics QA with Small-Model Adaptation. IEEE CSDE 2026. https://arxiv.org/abs/2610.00664 doi:10.48550/arxiv.2610.00664

q2 · i? · causal · r1

Results on the held-out test set (Table III): cloud baselines are zero-shot, edge models evaluated with and without LoRA. The authors explain cloud models generate longer, more explanatory responses than the shorter NCTB-standard phrases the lexical and semantic metrics reward.

"Cloud baselines have greater raw accuracy, 69.5 and 82.9 percent respectively but lower Token F1 and BERTScore than the adapted 4B model."

Discussion¶

Related Claims¶

• Fine-tuned 8B LLMs generate K-2 stories with difficulty-related metric scores statistically significantly different from (mostly better than) zero-shot GPT-4o and Llama 3.3 70B — related
• Model selection experiments show newer models are not strict improvements, with component-specific effects on quality metrics — related