The review's corpus comprises 39 peer-reviewed articles selected via a PRISMA protocol with substantial screening agreement¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? From 1196 initial manuscripts, screening and eligibility assessment using PRISMA procedures and inter-rater reliability checks (Cohen's Kappa 0.71; Fleiss' Kappa 0.55) yielded a final corpus of 39 articles. → Khaleel Asyraaf Mat Sanusi 2025

Evidence¶

Khaleel Asyraaf Mat Sanusi 2025¶

Khaleel Asyraaf Mat Sanusi, Deniz Iren, Nardie Fanchamps, Mai Geisen, Roland Klemke. (2025). Virtual virtuoso: a systematic literature review of immersive learning environments for psychomotor skill development. Education Tech Research Dev. https://doi.org/10.1007/s11423-025-10449-2

q2 · i? · quant-synthesis · r2

Methodology section describing the PRISMA-based selection: the initial search yielded 1196 manuscripts, reduced to 1177 after duplicate removal, with title/abstract screening achieving Cohen's Kappa of 0.71 and full-text eligibility voting achieving Fleiss' Kappa of 0.55, ultimately selecting 39 articles.

"Using Cohen's Kappa, a suitable measure for assessing agreement between two raters (Cohen, 1960), the first author and a co-author achieved a score of 0.71, indicating substantial agreement (Landis & Koch, 1977)."

Discussion¶

Related Claims¶

• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making it unsuitable for pass/fail decision consistency — related
• Rater selection via two-facet Rasch infit filtering substantially increases inter-rater reliability (Krippendorff's alpha) on ICNALE GRA — related
• The LoU Interview shows high interrater reliability, and the rating procedure was streamlined over two years without loss of agreement — related
• Expert human raters established an inter-rater reliability benchmark above ICC 0.89 on all four IELTS dimensions for the essay corpus — related
• Rubric-based assessment showed substantial to almost-perfect inter-rater reliability across all three time points — related
• Raters differed significantly in severity/leniency in the MFRM analysis — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• The corpus of the review comprised fifty-seven high-quality articles published between 2010 and 2017 that met stringent selection criteria — related