LLM-human factor-structure similarity is consistently lower than human-human similarity on both assessments¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Mean matched factor congruence for LLM-human comparisons fell reliably below the human-human resampling baseline on both instruments across factor solutions, with one-sided Wilcoxon rank-sum tests significant at p<.001 in all cases. → Alona Strugatski 2026

Evidence¶

Alona Strugatski 2026¶

Alona Strugatski, Licol Zeinfeld, Giora Alexandron. (2026). Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis. https://arxiv.org/abs/2608.15630 doi:10.48550/arxiv.2608.15630

q2 · i? · associational · r2

Repeated resampling analysis (100 iterations, matched 120-respondent samples) comparing HH and LH congruence distributions per instrument and factor number. Mean LH congruence was lower than HH on both instruments, and "this difference was significant (p<.001) in all cases"; no standardized effect size is printed in this passage.

"Averaged across all the tested factor solutions (4-8), mean similarity in Quantitative Reasoning was 0.535 for HH comparisons and 0.449 for LH comparisons; in Chemistry, the corresponding means were 0.595 and 0.513. One-sided Wilcoxon rank-sum tests confirmed that this difference was significant (p<.001) in all cases."

Discussion¶

Related Claims¶

• Item-to-factor loading patterns qualitatively differ between human and LLM EFA solutions — related
• Parallel analysis yields different factor counts for humans and LLMs on both instruments — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• Human-human agreement (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856) across constructs — related
• Self-report subscales do not reliably assess distinct metacognitive subcomponents, with structures of two to nine components found across studies — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related