Model and prompt choice account for only a small share of misalignment error, which concentrates in transcript-conditioned higher-order interactions¶

Claim · All claims
Evidence · 1 study · associational r3 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? A fully crossed Generalizability Theory decomposition shows LLM and PROMPT main effects explain little squared error (median shares 0.07 and 0.02), while higher-order interactions conditioning on classroom text dominate. → Michael Hardy 2026 q2 i? The dominant variance shares occur in interactions such as LLM×ITEM×OBS (0.19) and LLM×PROMPT×OBS (0.14), with a cell-specific remainder of 0.24. → Michael Hardy 2026

Evidence¶

Michael Hardy 2026¶

Michael Hardy, Yunsung Kim. (2026). Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact. arXiv. https://arxiv.org/abs/2603.00883 doi:10.48550/arxiv.2603.00883

q2 · i? · associational · r3

Fully crossed random-effects variance decomposition (Table 1) of squared misalignment error over OBS, ITEM, LLM, PROMPT facets. The abstract states model and/or prompting strategy "only reliably accounts for 15% of all measured misalignment error."

"the main effects of LLM and PROMPTexplain only a small portion of the squared-error landscape (median shares 0.07 and 0.02)"

Discussion¶

Related Claims¶

• Zero-shot prompt type has minimal impact on LLM-human coding concordance — related
• Prompt type interacts with coding dimension in error rates: definitions and instructions can impair detection of listing — related
• Prompt settings and temperature settings have significant main effects on correlations among the three LLM chatbots — related
• LLM-human agreement is highest for identifying whether students listed a concept and lowest for judging definition correctness — related
• In classroom video labeling, high raw agreement (often exceeding 90%) co-occurred with low chance-corrected κ due to temporal misalignment — related
• LLM-assigned difficulty labels fail to predict empirical item difficulty in classroom retrieval practice (ρ=0.06), as do co-generated Bloom levels (ρ=0.02) — related
• Off-the-shelf LLMs are systematically misaligned with human item-difficulty judgments and struggle to simulate lower-proficiency states — related
• During member checks, multilingual 8th-grade students contested both LLM classifications of their talk and the coding scheme itself, especially the Off-task and Claim codes — related
• Reasoning-capable model variants and chain-of-thought prompting yield no measurable improvement in either alignment axis for classroom evaluation — related
• GPT-4o and GPT-5.2 achieve higher accuracy and more fully correct answers on university IVDA multiple-choice questions than the smaller Mistral-7B and DeepSeek-R1 models — related
• Smaller local LLMs show a minor position bias toward option ordering and partially guessing-like error variation, but accuracy remains relatively stable across permutations and repeated sampling runs — related
• GPT-4o mini achieved the lowest ACT balance MAE (6.12) in replicating human supervisor ratings across 49 full transcripts — related