Five-part scope framework for post-exercise automated team assessment in TTXs¶
Research Method · All research methods
Evidence · 2 claims (2 for) · 1 study (1 design),q2· 0 of 1 report an effect size · 2 claims rest on one study
Description¶
The article defines its assessment scope through five grounded characteristics: assessment targets teams rather than individuals, since a team is the fundamental unit and succeeds only when members collectively resolve tasks; it is summative, using milestones as predefined checkpoints that serve as indicators of the learning objectives the team has met; it is exercise-specific, relying only on data collected during the TTX to protect privacy; it is post-exercise, delivering instant results within minutes; and it is fully automated, with manual assessment used only as an evaluation benchmark. This framework organizes the study's two methods and their educational purposes, including grading, hot-wash debriefing, and feedback on deviations from expected procedures.
Accounts¶
- Five-part scope framework for post-exercise automated team assessment in TTXs: The article defines its assessment scope through five grounded characteristics: assessment targets teams rather than individuals, since a team is the fundamental unit and succeeds only when members collectively resolve tasks; it is summative, using milestones as predefined checkpoints that serve as indicators of the learning objectives the team has met; it is exercise-specific, relying only on data collected during the TTX to protect privacy; it is post-exercise, delivering instant results within minutes; and it is fully automated, with manual assessment used only as an evaluation benchmark. This framework organizes the study's two methods and their educational purposes, including grading, hot-wash debriefing, and feedback on deviations from expected procedures. (V. Švábenský et al. (2026))
Claims¶
- Clustering of TTX activity logs aligns with instructor-assigned milestone scores and outperforms a baseline averaged over all possible groupings in both exercises [+M]
- GPT-4o's rubric-guided grading of team communication disagreed with instructors at near-chance levels (55% RMSE), while GPT-5.2 improved but remained insufficiently aligned (35%) [+M]
Related Research Methods¶
-
Key Sources¶
- V. Švábenský, J. Vykopal, S. Leelaluk, P. Čeleda, F. Okubo, A. Shimada. (2026). Assessment in Team Problem-Solving Exercises in Computing Education. Proceedings of the 56th IEEE Frontiers in Education Conference (FIE '26). https://arxiv.org/abs/2607.19209v1
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
five-part-scope-framework-for-post-exercise-automated-team-assessment-in-ttxs |
title |
Five-part scope framework for post-exercise automated team assessment in TTXs |
description |
The article defines its assessment scope through five grounded characteristics: assessment targets teams rather than individuals, since a team is the fundamental unit and succeeds only when members collectively resolve tasks; it is summative, using milestones as predefined checkpoints that serve as |
status |
review |
generated |
by process:settle-candidates on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.