Be cautious with LLM-based TTX assessment and have a human instructor review automated outputs¶

Strategy · All strategies
Evidence · no claims cited

Description¶

Based on their findings that general-purpose LLMs sometimes failed to recognize nuances in the domain-specific TTX context even when guided by a standardized rubric, the authors recommend caution about LLM-based assessment and state that, as with all automated decisions, the output must be reviewed by a human instructor to ensure accountability and fairness. They further note clustering's advantage: it considers only teams' activity logs, runs locally, and does not share learners' data with an external service, preserving privacy.

Design Implications¶

Context¶

Requirements¶

• A human instructor must review automated assessment outputs before they affect learners

Constraints¶

• Public (non-local) LLM assessment depends on an external service; if the LLM becomes unavailable or its version changes, the assessment's validity and reliability would change as well

Target Learners¶

• student teams in cybersecurity TTXs

Target Learning Goals¶

• fair and accountable automated assessment of team communication

Related Strategies¶

• Design TTXs with assessment in mind so that log data capture the learning process
• Automated assessment outputs must retain human monitoring and never serve as official grades or sole decision bases

Examples¶

-

Key Sources¶

• V. Švábenský, J. Vykopal, S. Leelaluk, P. Čeleda, F. Okubo, A. Shimada. (2026). Assessment in Team Problem-Solving Exercises in Computing Education. Proceedings of the 56th IEEE Frontiers in Education Conference (FIE '26). https://arxiv.org/abs/2607.19209