Success rates decrease monotonically from easy to medium to hard difficulty levels across all models¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? All models exhibit a clear monotonic trend, with success rates highest on easy tasks and decreasing on medium and hard tasks. → Chao Wen 2026

Evidence¶

Chao Wen 2026¶

Chao Wen, Jacqueline Staub, and Adish Singla. (2026). TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics. https://arxiv.org/abs/2606.03626 doi:10.48550/arxiv.2606.03626

q2 · i? · design · r2

Fine-grained analysis of symbolic success rates across the three labeled difficulty levels (Figure 5b) for representative VLMs. The article reports "a clear monotonic trend, with success rates highest on easy tasks and decreasing on medium and hard tasks", supporting the validity of the difficulty labeling. No effect size is printed.

"Across difficulty levels (see Figure 5b), all models exhibit a clear monotonic trend, with success rates highest on easy tasks and decreasing on medium and hard tasks, suggesting that the difficulty levels in our datasets capture meaningful differences in task complexity."

Discussion¶

Related Claims¶

• LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels — related
• Editing questions according to a hypothesis shifts measured difficulty in the expected direction — related
• Across frontier LLMs, Counseling is the easiest educational activity center (average 8.78) while Assessment and Research are hardest (8.42), and performance degrades from Easy to Hard difficulty — related
• Task difficulty increases hesitations, with significantly more hesitations on Medium and Hard tasks than Easy tasks — related
• Task difficulty increases error counts, with significantly more errors on Medium and Hard tasks than Easy tasks — related
• Goal setting improves performance — related
• LLM accuracy increases monotonically with prompt sophistication across semantically equivalent medical questions, with Low Literacy users scoring significantly below Prompt Engineer users — related
• Generated hypotheses discriminate hard from easy questions, mostly with medium-to-large effects — related