TURTLEAI benchmark

Product or Programme · All products and programmes
Evidence · 4 claims (2 for, 2 mixed) · 1 study (1 design), q2 · 0 of 1 report an effect size · 4 claims rest on one study

Description

A publicly released benchmark comprising Turtle Graphics programming-task datasets, code-evaluation tools, and synthetic-data generation methods, developed by Chao Wen et al. for evaluating vision-language models on educational visual programming.

Components

Claims

• Current vision-language models struggle on education-oriented Turtle Graphics tasks, with even top models far below reliable performance on real-world tasks for grades 3-6 [+M]
• Fine-tuning VLMs on synthetic data from TURTLEAI-Datagen improves performance on TURTLEAI, raising symbolic success rates by over 10% over base counterparts [+M]
• Fine-tuning primarily improves alignment between code generation and visual reasoning rather than visual understanding or spatial reasoning [~W]
• All evaluated models struggle with spatial reasoning and precise visual replication on Turtle Graphics tasks [~W]

Related Products and Programmes

• MEDS dataset of 28,000 LLM runs across four mathematics-education tasks
• The Pedagogy Benchmark

Key Sources

• Chao Wen, Jacqueline Staub, and Adish Singla. (2026). TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics. https://arxiv.org/abs/2606.03626

Page metadata

Field	Value
type	product
id	turtleai-benchmark
title	TURTLEAI benchmark
description	A publicly released benchmark comprising Turtle Graphics programming-task datasets, code-evaluation tools, and synthetic-data generation methods, developed by Chao Wen et al.
status	review
product_kind	dataset
generated	by process:settle-candidates on 2026-10-12
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.