AI pilots found models not yet capable of high-quality math output generation, including multiple choice problems, coherent word problems, and math visuals¶

Claim · All claims
Evidence · 1 study · qualitative r? · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? Cohort teams found AI could not reliably generate meaningful multiple choice problems, coherent word problems, or math visuals such as charts and graphs. → math-output

Evidence¶

math-output¶

Noakes, S., Shell, A., Murillo, A., Van Nostrand, P., Ruiz, P., Cornick, S., and Karim, S. (2024, October). An Ethical and Equitable Vision of AI in Education: Learning Across 28 Exploratory Projects. Digital Promise. https://doi.org/10.51388/20.500.12265/232

q2 · i? · qualitative · r?

Technical findings from the 28-project cohort, summarized in Table 5. Teams reported that in some instances models "generated four options that were all wrong or all right" and produced word problems incoherent with the concepts considered; generating math visuals was also not usable during the projects.

"This is an area where models are currently not technically capable of high-quality outputs. In particular, AI (or at least ChatGPT4.0 as of spring 2024) is not yet able to generate meaningful multiple choice problems"

Discussion¶

Related Claims¶

• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
• AI was found lacking in diagnosing student misconceptions because models do not understand lack of logic or sense making — related
• Mathematics teachers critique ChatGPT-generated generative learning lessons for poor fit to students, boredom, preparation time, missing student-facing materials, mathematical flaws, and required materials — related
• Three word embedding methods and three classifiers were introduced to predict item quality for accessible math assessments — related
• Generic AI tools can undermine curriculum coherence and pedagogical goals, producing low-quality artifacts that do not match district beliefs — a broader claim this one bears on
• Differences between AI-generated and teacher assessments of laboratory reports may stem from difficulties retrieving and interpreting equations, graphs, tables, and units, not only from model reasoning capabilities. — related
• The most critical challenge in integrating AI into instructional design is pedagogical alignment (14 of 28 studies, 50.0%) — a broader claim this one bears on
• Visual media assessments (posters, presentations) were not passable because ChatGPT could not produce complete slides or poster content — related
• Across 28 K-12 AI pilot projects, AI demonstrated success at categorizing and coding datasets, translating language, and triangulating qualitative findings with quantitative data — related