VLMs show a common grounding failure: coarse semantic relevance without precise region-activity binding and entity-evidence alignment¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Error analysis across Activity Localization, Activity Description, and Remote Interaction reveals that models capture coarse semantic relevance but accept plausible relations without verifying joint alignment of entities, regions, and evidence. → Luyao Zhu 2026

Evidence¶

Luyao Zhu 2026¶

Luyao Zhu, Xun Wei Yee, Wei Li, Mun Thye Mak, Wee Siong Ng. (2026). MUSE: Benchmarking Large Vision-Language Models on Multi-modal Understanding in Situated Education. arXiv preprint. https://arxiv.org/abs/2609.19088 doi:10.48550/arxiv.2609.19088

q2 · i? · design · r2

Cross-task error analysis of incorrect option and evidence-selection type distributions across models for three tasks (Figure 8). The article reports that in Activity Localization models "usually select semantically relevant people or activities rather than random regions, but fail to identify the complete target extent."

"In Remote Interaction, mismatched region-text pairs dominate, showing that models often accept plausible relations without verifying whether entities, regions, and evidence are jointly aligned."

Discussion¶

Related Claims¶

• MLLMs show recurring failures in fine-grained quantitative estimation, often identifying the encoding and measurement strategy but misestimating precise values — a narrower finding that bears on this claim
• Some VLMs perform near chance on binary judgments of student correctness, and error-assessment behavior is idiosyncratic across models — related
• Human performance forms the outer envelope on nearly all MUSE tasks, with the largest deficits in Relative Position, Emotion Detection, and Jigsaw Puzzle — related
• VLM performance on artistic educational imagery is highly task-dependent, with no model dominating across capabilities — related
• Some MLLM failures occur when a higher-level SciVis interpretation is grounded in an unsupported visual encoding rather than the encoding specified by the visualization and caption — a narrower finding that bears on this claim