GPT-4o mini showed progressive turn-level convergence with accumulating context while larger models showed increasing or stable error¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? In partial-transcript evaluation, GPT-4o mini's MAE decreased from 12.76 at turn 5 to 6.12 at turn 20, whereas GPT-5.1, GPT-5.2, and Claude Sonnet 4.5 showed increasing or stable error with additional context. → Pascal Riachi 2026

Evidence¶

Pascal Riachi 2026¶

Pascal Riachi, Sofie Kamber, Stella Brogna, Andrew Gloster, Rafael Wampfler. (2026). Toward Accessible Psychotherapy Training Using AI-Driven Interactive Patient Avatars. https://arxiv.org/abs/2606.17786 doi:10.48550/arxiv.2606.17786

q2 · i? · design · r2

Turn-level evaluation rating partial transcripts consisting of the first t therapist turns, computing MAE between the model's partial-transcript rating and the human supervisor's full-session rating. GPT-4o mini showed "progressive alignment" with a 6.64-point MAE improvement from turn 5 to turn 20.

"GPT-4o mini showed progressive alignment, with MAE decreasing from 12.76 at turn 5 to 6.12 at turn 20 (6.64-point improvement)."

Discussion¶

Related Claims¶

• Expert evaluation found the virtual patient realistic and immediate ACT feedback increased therapists' awareness of intervention choices — related
• Model selection experiments show newer models are not strict improvements, with component-specific effects on quality metrics — related
• GPT-4o mini achieved the lowest ACT balance MAE (6.12) in replicating human supervisor ratings across 49 full transcripts — related
• GPT-4o's rubric-guided grading of team communication disagreed with instructors at near-chance levels (55% RMSE), while GPT-5.2 improved but remained insufficiently aligned (35%) — related
• GPT-4o-mini natural language feedback was overwhelmingly helpful and never harmful in quality annotation — related
• Models with similar rank correlations differ substantially in estimating difficulty magnitude (RMSE and MAE) — related
• LLM ratings of classroom transcripts are more correlated with each other than with expert human ratings, across the same and different tasks — related