GPT-4 outperforms fine-tuned T5 by a normalized accuracy improvement of 5.73% for dialogue act recognition on the same held-out test set.¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The GPT-4 model outperformed the previous state-of-the-art T5 model with a normalized accuracy improvement of 5.73% on the held-out test set. → Spain 2024

Evidence¶

Spain 2024¶

Spain, R., Min, W., Kumaran, V., Pande, V., Saville, J., & Lester, J. (2024). Applying Large Language Models to Enhance Dialogue and Communication Analysis for Adaptive Team Training. https://doi.org/10.21203/rs.3.rs-4565500/v1

q2 · i? · design · r2

Comparison of the highest-performing GPT-4 model against the T5 baseline from prior work on the same held-out test set. The article reports that "the GPT-4 model outperformed the previous state-of-the-art models using T5 (Pande et al., 2023) with a normalized accuracy improvement of 5.73%".

"Notably, the GPT-4 model outperformed the previous state-of-the-art models using T5 (Pande et al., 2023) with a normalized accuracy improvement of 5.73%."

Discussion¶

Related Claims¶

• Zero-shot GPT-4 with iterative prompt engineering achieves 77.30% dialogue act recognition accuracy, outperforming fine-tuned T5, CRF, and BLSTM baselines on a held-out test set. — possibly the same claim (merge candidate)
• Prompt refinements targeting inter-label confusion yield superior recall rates for challenging dialogue act labels, including Command (F1 raised to 0.75) and Other (GPT-4 recall 0.67 vs. T5 0.54). — related