Jonassen-informed prompt-variation framework with MAPS-based evaluation for probing LRM physics reasoning¶
Research Method · All research methods
Evidence · 3 claims (2 for, 1 mixed) · 2 studies (2 causal),q2· 0 of 2 report an effect size · 3 claims rest on one study
Description¶
The article adapts Jonassen's problem-classification framework to generative-AI problem solving along two dimensions: "prompt-specificity, or how much information is explicitly given versus left for the model to infer, and modality, or whether the problem is presented through text alone or with visual representations". Combined with the MAPS rubric, which scores useful descriptions, physics approach, specific application, mathematical procedures, and logical progression, this provides "a structured way to probe these dimensions of LRM reasoning" beyond final-answer accuracy.
Accounts¶
- Jonassen-informed prompt-variation framework with MAPS-based evaluation for probing LRM physics reasoning: The article adapts Jonassen's problem-classification framework to generative-AI problem solving along two dimensions: "prompt-specificity, or how much information is explicitly given versus left for the model to infer, and modality, or whether the problem is presented through text alone or with visual representations". Combined with the MAPS rubric, which scores useful descriptions, physics approach, specific application, mathematical procedures, and logical progression, this provides "a structured way to probe these dimensions of LRM reasoning" beyond final-answer accuracy. (Nikhil Sanjay Borse et al. (2026))
Claims¶
- Only the well-specified text-only prompt produced a correct quantitative result from o4-mini, while underspecified prompts led to numerical errors and misleading final results [+M]
- Multimodal prompts caused o4-mini to skip numerical integration and produce only qualitative final responses despite recovering much of the analytical structure [+M]
- Three publicly available LLMs achieve 96% average accuracy on text-only physics problems with no significant differences among models [~W]
Related Research Methods¶
-
Key Sources¶
- Nikhil Sanjay Borse, Amir Bralin, Sean Savage, N. Sanjay Rebello. (2026). Probing AI-generated physics solutions and preparing students to critique them. https://arxiv.org/abs/2608.12533
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
jonassen-informed-prompt-variation-framework-with-maps-based-evaluation-for-probing-lrm-ph |
title |
Jonassen-informed prompt-variation framework with MAPS-based evaluation for probing LRM physics reasoning |
description |
The article adapts Jonassen's problem-classification framework to generative-AI problem solving along two dimensions: "prompt-specificity, or how much information is explicitly given versus left for the model to infer, and modality, or whether the problem is presented through text alone or with visu |
status |
review |
generated |
by process:settle-candidates on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.