Match LLM temperature settings to construct characteristics when coding educational dialogue¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends calibrating LLM sampling parameters to the psychological construct being coded rather than applying one configuration universally. Its results suggest "constructs with moderate theoretical coherence benefited from higher temperatures, while well-defined constructs required deterministic settings," and that low-clarity constructs may need higher temperatures to capture varied expressions. Without systematic frameworks for matching parameter configurations to construct types, researchers risk settings that bias results toward certain constructs while undermining the reliability of others.

Design Implications¶

Context¶

Requirements¶

• Systematic evaluation to ensure methodological choices produce outcomes likely to be reliable and interpretable by researchers; construct definitions and prompts refined with clear operational definitions and examples of evidence

Constraints¶

• Even setting temperature to 0 does not guarantee perfect reproducibility, as model updates, non-deterministic software elements, and API-specific configurations can still produce variations across sessions and platforms

Target Learners¶

• middle school students classified as English learners

Target Learning Goals¶

• reliable automated coding of persistence and related psychological constructs from student dialogue

Affordances¶

• Construct Evaluation Framework Measurement Dimensions

Related Strategies¶

• Use structured multi-stage prompts with clearly defined constructs when eliciting LLM topic labels for qualitative coding
• Use deterministic, fully replicable text-analysis methods with documented search terms when analyzing large text corpora

Examples¶

-

Key Sources¶

• Ober, T. M., Zhang, S., Zapata-Rivera, D., Schroeder, N. L., & Botelho, A. F. (2026). Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent. Journal of Educational Data Mining, Volume 18, No 1. https://doi.org/10.17605/osf.io/s85ck