Design CT rubrics and prompts to explicitly operationalize integrated constructs and demand parameter-interaction reasoning¶

Strategy · All strategies
Evidence · no claims cited

Description¶

Based on the finding that both humans and the LLM were least reliable on multi-component constructs, the authors recommend rubric and prompt revisions. They write that "Future rubrics should de- fine STP more explicitly, distinguishing between identifying system components and explaining their interactions and consequences," and that future assessment versions "should ask students to explain how parameters interact within the system" rather than merely listing factors. They also caution that LLM-based CT assessment requires careful rubric design and human oversight for complex reasoning.

Design Implications¶

Context¶

Requirements¶

• Rubrics must define Systems Thinking explicitly, separating component identification from explanation of interactions and consequences

Constraints¶

• LLM-based scalable assessment requires careful rubric design and human oversight for complex reasoning

Target Learners¶

• students in large-enrollment introductory physics courses

Target Learning Goals¶

• reliable assessment of computational thinking, especially integrated constructs like systems thinking

Related Strategies¶

• Define correctness explicitly in LLM-as-a-judge prompts, offering behavior-focused and construction-focused rubric variants

Examples¶

-

Key Sources¶

• Sean Savage, Anand Shanker, Grace Michlitsch, and N. Sanjay Rebello. (2026). Using LLMs to Detect Growth in Computational Thinking in Introductory Physics. arXiv preprint. https://arxiv.org/abs/2608.06200