Same-center cross-organ handbook agreement exceeds different-center same-organ agreement by 0.024, but the institution contrast is sensitive to coverage thresholds and document selection¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Same-center cross-organ agreement exceeds different-center same-organ agreement by 0.024 (Holm-adjusted p = 0.011) in the primary analysis of handbook pairs. → Yubo Li 2026 q2 i? The contrast is sensitive to document selection: restricting to complete short-handbook text leaves the key contrast at 0.002 with a confidence interval spanning zero. → Yubo Li 2026 (2)

Evidence¶

Yubo Li 2026¶

Yubo Li, Rema Padman, Ramayya Krishnan. (2026). Auditing Institutional Heterogeneity for Generative AI in Patient Education: A Large-Scale Study of 102 US Transplant Handbooks. arXiv preprint. https://arxiv.org/abs/2607.22606 doi:10.48550/arxiv.2607.22606

q2 · i? · associational · r2

Exploratory Welch t-test comparison across 2,670 handbook pairs with at least 20 non-absent comparisons, stratified by shared center and organ in the production run. The article reports the printed contrast and Holm-adjusted p-value, calling the tests "exploratory because pairs share handbooks."

"Same-center cross-organ agreement exceeds different-center same-organ agreement by 0.024 (95% Welch CI [0.007,0.040] ; t= 2.82 ; raw p= 0.0056 , Holm-adjusted p= 0.011 )."

Yubo Li 2026 (2)¶

Yubo Li, Rema Padman, Ramayya Krishnan. (2026). Auditing Institutional Heterogeneity for Generative AI in Patient Education: A Large-Scale Study of 102 US Transplant Handbooks. arXiv preprint. https://arxiv.org/abs/2607.22606

q2 · i? · associational · r2

Sensitivity analysis on answers generated from complete short-handbook text, which removes passage-selection differences but changes corpus composition. The contrast drops to 0.002 with a CI spanning zero, so the article concludes the institution contrast is sensitive to coverage and document selection.

"Restricting to answers generated from complete short-handbook text leaves 479 qualifying handbook pairs; the key contrast is 0.002 (95% CI [−0.034,0.037] ; 23 samecenter cross-organ versus 194 different-center sameorgan pairs)."

Discussion¶

Related Claims¶

• Multimodal LLMs show moderate cross-model agreement on which image-Required items are solvable, with within-family agreement exceeding cross-family agreement — related
• Reproductive health shows the highest coverage gap (82.1% absence) and the highest judge-rated significance among divergent pairs (86%), making it a priority content area — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related