Allocate human and AI effort by dimension type: automate explicit-criteria dimensions, retain human review for pedagogical judgment¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article's central design recommendation is a principled division of labor: agentic AI with retrieval grounding and computational tools can serve as scalable first-line quality control for dimensions grounded in explicit criteria and verifiable correctness (clarity, code validity, concept alignment, correct answer validity), while human experts remain essential for pedagogical depth, meaningful misconception targeting in distractors, and pedagogically rich feedback. The article states these findings "inform the strategic allocation of human and AI effort in AI-assisted educational content generation."

Design Implications¶

Context¶

Requirements¶

• Systematic validation of automated evaluator performance against human expert judgment rather than assuming evaluator reliability

Constraints¶

• Automated validation was prone to approving items that were technically correct but instructionally shallow

Target Learners¶

• introductory programming students

Target Learning Goals¶

• code reasoning and comprehension
• high-quality assessment item generation

Affordances¶

• Code Gen Dual Agent Architecture

Related Strategies¶

• Use accuracy-rejection curves with reliability analysis to select operating points for selective delegation in human-in-the-loop grading
• Use automated collaboration assessments as scaffolds for reflection rather than ground truth
• Use observable pedagogical variables as metadata to support teachers' explicit selection and integration of digital resources
• Deploy AI-assisted classroom observation as a complementary first-pass screening and reflection tool, with raters retaining interpretive judgment
• Structure AI-assisted grading to shift TA effort from routine low-judgment decisions toward conceptually rich responses
• Treat AI-generated lesson plans as drafts requiring teacher evaluation, adaptation, and pedagogical judgment
• Human-in-the-loop validation workflow: hold all AI-generated teaching content in unreviewed status until a teacher manually approves it
• Use automatic reflection classification for preliminary feedback and workload support, with final assessment retained by educators
• Use an AI-assisted vibe coding workflow with human pedagogical control to move educational tools from idea to classroom within days
• Implement AI in assessment as a hybrid support tool for low-stakes, formative stages while humans retain final grading authority
• Use AI chatbots to assist human-led coaching with routine support between sessions while humans retain challenge and goal clarification
• Design assignments that explicitly specify which tasks may be delegated to AI and which may not
• Evaluate LLM data extraction through a staged meaningful-human-control sequence before integrating it into evidence synthesis

Examples¶

-

Key Sources¶

• Xiaojing Duan, Frederick Nwanganga, and Chaoli Wang. (2026). CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation. https://arxiv.org/abs/2604.03926