Human expert review is the most credible but slowest and most resource-intensive evaluation method¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Among evaluation methods, human expert review is the most credible but also the slowest and most resource-intensive, and remains the foundation for defining quality and consistency. → Li 2026

Evidence¶

Li 2026¶

Li, W., Kurimchak, M., & Whitmer, J. (2026). ATS Field Note: AI Model Evaluation. ATS Hub Field Note Series. https://doi.org/10.51388/20.500.12265/300

q2 · i? · theoretical · r2

The field note's spectrum of evaluation methods characterizes human expert review as "the most credible, but also the slowest and most resource-intensive". This is the authors' framework judgment, not a measured comparison.

"Human expert review is the most credible, but also the slowest and most resource-intensive. It remains the foundation for defining quality and consistency."

Discussion¶

Related Claims¶

• Instructional design models are rarely tested against outcomes; their credibility comes from practitioners finding them useful. — related
• One-session checklist (CRAAP) instruction in source evaluation did no better than the alternative methods it was compared with in two quasi-experiments — related
• The expert-designed KC model adds little predictive power on most datasets, with significant contributions only on the two KDD Cup 2010 datasets — related
• LLM-as-judge automated evaluation can achieve human-level agreement when carefully validated — reports the opposite
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related