LoRA adaptation significantly increases closed-book accuracy on the PhysicsMate test set at all three Qwen3 scales¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Fine-tuning with a unified LoRA recipe improves every metric at every scale, with accuracy gains of +5.5, +15.0 and +23.3 percentage points at 0.6B, 1.7B and 4B. → Rashid Azraf Jahin 2026

Evidence¶

Rashid Azraf Jahin 2026¶

Rashid Azraf Jahin, Saadman Sajid, Khan Raiyan Ibne Reza, Sumaiya Tabassum Nimi. (2026). PhysicsMate: A Curriculum-Grounded Bengali Benchmark for Secondary Physics QA with Small-Model Adaptation. IEEE CSDE 2026. https://arxiv.org/abs/2610.00664 doi:10.48550/arxiv.2610.00664

q2 · i? · causal · r1

Evaluation on the 275 held-out test questions with greedy decoding, LLM-judge accuracy (Gemini-2.5-Flash-Lite). The 1.7B model's accuracy "is doubled, from 10.5% to 25.5%", and the 4B model reaches 50.9% fine-tuned versus 27.6% raw. No effect sizes are printed.

"Fine-tuning improves every metric at every scale. The base model, Qwen3-0.6B, is not able to get the correct answer rate on the test set, but following adaptation, the rate becomes 5.5%"

Discussion¶

Related Claims¶

• Adaptation gains grow monotonically with model capacity under the same LoRA recipe, though three data points cannot establish a scaling law — a broader claim this one bears on
• LoRA fine-tuning on control systems course data increases lexical and semantic similarity between model responses and course reference answers at both 3B and 7B model sizes — related
• Base-model size effects persist after LoRA adaptation: the 7B model gained more on ROUGE-2 and ROUGE-L, yet 3B-r16 outperformed 7B-r4 on ROUGE-L and BERTScore-F1 — related
• Specializing models for K-12 education does not require a broad loss of general capability, with MMMU-Pro accuracy increasing at all three scales — related
• Education-oriented instruction tuning consistently improves curriculum grounding, K-12 problem solving, and pedagogical tutoring at every evaluated model scale — related
• Adaptation benefits structured curricular knowledge (quantities, laws, definitions) most and loosely specified entity-level knowledge least — related