All evaluated models struggle with spatial reasoning and precise visual replication on Turtle Graphics tasks¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? All models struggle with spatial reasoning — reasoning about relative positions, distances, angles, and sizes of patterns in the image. → Chao Wen 2026 q2 i? Despite correct visual reasoning, models fail to achieve low-level visual accuracy, ignoring tiny details like angles and relative positions. → Chao Wen 2026 (2)

Evidence¶

Chao Wen 2026¶

Chao Wen, Jacqueline Staub, and Adish Singla. (2026). TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics. https://arxiv.org/abs/2606.03626 doi:10.48550/arxiv.2606.03626

q2 · i? · design · r2

Systematic manual failure analysis on DSReal for GPT-4o, Qwen2-VL-72B, and Qwen2-VL-72B-TURTLE, attributing each failure case to a failure type. The analysis found "all models struggle with spatial reasoning"; the authors suggest this might be due to scarcity of training data capturing spatial relationships. No effect size is printed.

"We find that all models struggle with spatial reasoning, which is the ability to reason about the spatial relationships among different patterns in the image, such as relative positions, distances, angles, and sizes of patterns."

Chao Wen 2026 (2)¶

Chao Wen, Jacqueline Staub, and Adish Singla. (2026). TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics. https://arxiv.org/abs/2606.03626

q2 · i? · design · r2

Same manual failure analysis on DSReal: models can often replicate the intended image from a high-level perspective but fail low-level visual accuracy, such as ignoring tiny details like angles and relative positions. No effect size is printed.

"We find that despite correct visual reasoning, models still face difficulties in achieving visual precision during replication."

Discussion¶

Related Claims¶

• Fine-tuning is associated with more compact generated code, with the fine-tuned Qwen2-VL-72B maintaining a length ratio near 1.0 while the base model's failed outputs become substantially longer — related
• Fine-tuning primarily improves alignment between code generation and visual reasoning rather than visual understanding or spatial reasoning — related
• Current vision-language models struggle on education-oriented Turtle Graphics tasks, with even top models far below reliable performance on real-world tasks for grades 3-6 — related
• Back-translation yields higher human agreement on spatial checks, while LLM-as-a-Judge considerably outperforms it on the angle-label check — related
• All evaluated GenAI systems scored noticeably lower on graphics-related questions requiring interpretation of JavaFX output — related
• Viewpoint-aware spatial reasoning is a persistent bottleneck with poor calibration of spatial uncertainty — a broader claim this one bears on
• Fine-tuning VLMs on synthetic data from TURTLEAI-Datagen improves performance on TURTLEAI, raising symbolic success rates by over 10% over base counterparts — related
• Fine-tuning on clean synthetic data yields limited gains on hand-drawn tasks, indicating limited robustness to out-of-distribution hand-drawn inputs — related