The CALM tool showed only weak inter-rater agreement (weighted Kappa 0.45) in its single ED video-review study¶
Claim · All claims
Evidence · 1 study · reviewr3·q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Kava et al.'s weighted Kappa of 0.45 between two experts is characterised by the review as a weak level of agreement. → Alexander Emily G 2023
Evidence¶
Alexander Emily G 2023¶
Alexander Emily G, Denny Fraser, Gordon Malcom WG, McKiernan Cieran, Lowe David J. (2023). Evaluation of video review tools for assessing non-technical skills in emergency department resuscitation teams: a systematic review. BMC Emergency Medicine. https://doi.org/10.1186/s12873-023-00895-7
q2 · i? · review · r3
The review reports the single CALM study, a prospective randomised controlled pilot with 20 patients in which residents reviewed performance between resuscitations; agreement was "0.45 (CI 0.35–0.56, p < 0.0001)", judged weak. CALM assesses 15 NTS components versus T-NOTECHS's 5.
"The CALM paper, Kava et al [24], used weighted Kappa between two experts to ensure agreement and it was 0.45 (CI 0.35–0.56, p < 0.0001). This is a weak level of agreement."
Discussion¶
Related Claims¶
- An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
- Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality — related
- NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — related
- A systematic review of ED resuscitation video review identified four unique NTS assessment tools, with T-NOTECHS used in six of nine studies — a broader claim this one bears on
- Incorporating video review feedback into leadership training had a positive effect on emergency medicine residents' team-leader performance in a pilot study — related
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.