Gold and ultra-gold benchmark subsets¶
Product or Programme · All products and programmes
Evidence · no claims cited
Description¶
Croteau et al. released reproducible gold and ultra-gold subsets of image-required middle-school mathematics items for evaluating multimodal language-model accuracy, stability, refusal, and description-substitution performance.
Components¶
- Gold and ultra-gold benchmark subsets of consistently solved image-Required items: The article identifies and releases two reproducible benchmark subsets of image-Required items: "61ultra-golditems solved 3/3 by all six models and 76golditems solved by all six at majority-correct". These subsets support future diagnostics and comparisons beyond single-shot accuracy, and the authors propose them as a testbed for description-substitution experiments comparing original figures to vetted textual descriptions while monitoring accuracy, stability, and refusal. Code, prompts, and summary artifacts are released at the stated OSF link. (Croteau et al. (2026))
Claims¶
Related Products and Programmes¶
Key Sources¶
- Croteau, E., & Heffernan, N. (2026). Seeing Is Solving: MLLMs, Reasoning, and Refusal in Visual Math. https://osf.io/ct7bg/
Page metadata
| Field | Value |
|---|---|
type |
product |
id |
gold-and-ultra-gold-benchmark-subsets |
title |
Gold and ultra-gold benchmark subsets |
description |
Croteau et al. |
status |
review |
moved_from |
element/gold-ultra-gold-benchmark-subsets |
product_kind |
dataset |
generated |
by process:sweep-kinds on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — croteau-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.