LLM-as-judge evaluation scored CyberAGENTS highest on domain grounding (4.46) and agent role fidelity (4.35)¶

Claim · All claims
Evidence · 1 study · design r2 · q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? Across 74 recorded student interactions, the LLM-as-judge assigned the highest scores to Domain Relevance and Cybersecurity Grounding (4.46) and Agent Role Fidelity (4.35) on a 1-5 scale. → Hornung 2026

Evidence¶

Hornung 2026¶

Hornung, I., Marasinghe Arachchige, D., Kumarage, T., Agrawal, G., Deng, Y., Chen, Y.-C., & Liu, H. (2026). CyberAGENTS: Structured Autonomy for Agentic Gamified Learning in Cybersecurity. Preprint. https://arxiv.org/abs/2608.07965 doi:10.48550/arxiv.2608.07965

q1 · i? · design · r2

LLM-as-judge evaluation of complete transcripts using category-specific rubric prompts, over 74 recorded student interactions for the full system. Scores are on a 1-5 scale; Interaction Coherence and State Tracking was somewhat lower at 3.95, which the rubric labels "Mostly coherent" behavior.

"The highest scores are obtained in Domain Relevance and Cybersecurity Grounding (4.46) and Agent Role Fidelity (4.35), as seen in Table 3"

Discussion¶

Related Claims¶

• Ablating expert-informed structural components lowered LLM-judge scores on all dimensions, with largest drops in challenge quality, intent alignment, and domain grounding — related
• Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related