Human performance forms the outer envelope on nearly all MUSE tasks, with the largest deficits in Relative Position, Emotion Detection, and Jigsaw Puzzle¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? In a sampled human-model comparison, the human average forms the outer performance envelope on nearly all 12 tasks; the largest human-VLM deficits occur in Relative Position, Emotion Detection, and Jigsaw Puzzle, and no model family consistently approaches human performance across all capabilities. → Luyao Zhu 2026

Evidence¶

Luyao Zhu 2026¶

Luyao Zhu, Xun Wei Yee, Wei Li, Mun Thye Mak, Wee Siong Ng. (2026). MUSE: Benchmarking Large Vision-Language Models on Multi-modal Understanding in Situated Education. arXiv preprint. https://arxiv.org/abs/2609.19088 doi:10.48550/arxiv.2609.19088

q2 · i? · design · r2

Human-model comparison in which 20 questions were randomly sampled from each of the 12 tasks and answered by two annotators, compared with model responses on the same questions (Figure 11). The article reports the gap is narrower for Activity Localization, Object Count, Remote Interaction, and Scene Classification.

"Figure 11 shows that the human average forms the outer performance envelope on nearly all tasks, demonstrating a substantial gap between current VLMs and human multimodal understanding."

Discussion¶

Related Claims¶

• A recognition-versus-reasoning divide: scene recognition is comparatively mature while affective interpretation and compositional reasoning lag — related
• VLM performance on artistic educational imagery is highly task-dependent, with no model dominating across capabilities — related
• VLMs show a common grounding failure: coarse semantic relevance without precise region-activity binding and entity-evidence alignment — related
• MUSE tasks form coherent but non-redundant capability groups, with Jigsaw Puzzle a distinct compositional capability — related