Hungarian transformer models achieve slightly lower overall score (about 68%) but better minority-class sensitivity than shallow models¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Transformer models reach about 68% overall score while classifying minority reflection classes better than shallow baselines, though most minority samples are still misclassified. → Zsolt Csibi 2026

Evidence¶

Zsolt Csibi 2026¶

Zsolt Csibi, Kristian Fenech, Mónika Sándor, Mónika Serfőző, Kinga Gyöngy. (2026). Automatic Reflection Level Classification in Hungarian Student Essays. https://arxiv.org/abs/2605.02402 doi:10.48550/arxiv.2605.02402

q2 · i? · design · r2

Fine-tuning experiment of two Hungarian transformers (hubert-base-cc, PULI-BERT-Large) on the essay dataset with stratified 5-fold cross-validation. Confusion matrices in Fig. 5c and 5d show "improved minority-class detection for transformer models compared to shallow baselines in these settings."

"The best results with the transformer models are around≈68% overall score averaged over accuracy, F1-score, and ROC AUC metrics. However the transformer models are able to classify the minority classes better, as the confusion matrix in Fig. 5c and Fig. 5d presents."

Discussion¶

Related Claims¶

• Shallow machine learning with SMOTE oversampling and Qwen3 embeddings achieves the best overall reflection-classification performance (0.7176) on Hungarian essays — related
• T-tests show no significant shallow-vs-transformer difference in accuracy, weighted F1 or macro-F1, but a significant ROC-AUC advantage for the shallow model — reports the opposite
• For transformer models, oversampling and backtranslation augmentation improve performance while freezing the encoder substantially decreases it — related