Fully inductive LLM codebook development risks importing unexamined sensitizing concepts, such as folk theories and scientific misconceptions, from the model's training data¶

Claim · All claims
Evidence · 1 study · design r3 · q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? If not prompted toward specific theory, GPT may implicitly apply sensitizing concepts drawn from non-neutral training data, and such implicit theoretical choices may go unnoticed by researchers. → Zambrano 2026

Evidence¶

Zambrano 2026¶

Zambrano, A.F., Wei, Z., Zhang, J., Baker, R.S., Ocumpaugh, J., Barany, A., Liu, X., Ginger, J., Paquette, L., Zhou, Y., & Borchers, C. (2026). Data Plus Theory Equals Codebook: Leveraging LLMs for Human-AI Codebook Development. Journal of Educational Data Mining, Volume 18, No 1. https://osf.io/g3z4x doi:10.35542/osf.io/4wyqc_v1

q1 · i? · design · r3

Theoretical argument, not a tested result: the authors reason from prior work that LLM training data "is not neutral or atheoretical", so fully inductive prompting risks unnoticed biases that expert oversight may miss, such as missing codes or blends of related theories.

"If not prompted to consider specific theory, GPT might draw its sensitizing concepts from, for example, the folk theories or widely believed scientific misconceptions present in its training data"

Discussion¶

Related Claims¶

• GPT-4o prompted only with SRL theory references generates a codebook accurately representing most key elements of both foundational SRL theories — related
• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — a broader claim this one bears on
• A hybrid human-AI workflow using GPT-4o with retrieval-augmented generation supported efficient inductive thematic analysis while preserving researcher judgment — related
• In the interest-development context, naming the theory without full references produced the most practical and usable codebook, while supplying full papers enhanced theoretical alignment but reduced applicability — related
• Adding think-aloud data to theory prompts improves GPT-4o codebook completeness and alignment with the data — related
• Five common AI misconceptions debunked: AI does not think like humans, create original ideas, replace teachers, or guarantee accuracy, and well-designed AI use need not lower rigor — related
• Attachment to LLMs exhibits cruel optimism: desired efficiency collides with the vigilance and expertise students do not yet possess — related
• A general-purpose LLM assessing team emails' emotional tone was biased toward interpreting messages as anxiety-related only — a narrower finding that bears on this claim
• Prompting is a form of writing: task definition is a constitutive element of knowledge production, not a neutral procedural step — a broader claim this one bears on
• A three-phase human-LLM inductive pipeline produced a hierarchical codebook of 72 items within 19 categories and six domains from 45,000 educator-AI messages — related
• All CSCL research is necessarily theory-laden; unreflective research adopts commonsense folk theories — related
• Teachers hold a complicated mix of scientific conceptions, misconceptions, and uncertainty about dyslexia concepts — related
• Human review remains essential after GPT-based codebook generation, producing a final refined SRL codebook of eleven constructs — related
• GPT-5 used as a third coder introduced a calculation_check category absent from the human codebook, covering 28% of messages and exposing an unnamed recurring request type — related