Human review remains essential after GPT-based codebook generation, producing a final refined SRL codebook of eleven constructs¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Two human coders merged overlapping constructs, renamed codes, and excluded inapplicable ones, reducing the GPT-generated codebook to eleven constructs. → Zambrano 2026

Evidence¶

Zambrano 2026¶

Zambrano, A.F., Wei, Z., Zhang, J., Baker, R.S., Ocumpaugh, J., Barany, A., Liu, X., Ginger, J., Paquette, L., Zhou, Y., & Borchers, C. (2026). Data Plus Theory Equals Codebook: Leveraging LLMs for Human-AI Codebook Development. Journal of Educational Data Mining, Volume 18, No 1. https://osf.io/g3z4x doi:10.35542/osf.io/4wyqc_v1

q2 · i? · design · r2

Step 3 human refinement by two coders evaluated specificity and applicability of GPT's constructs, collapsing overlapping codes (e.g., Planning and Goal Setting) and renaming Calculation & Conceptual Understanding as Domain Knowledge Recall. The authors conclude "human review remains essential for code merging and codebook refinement".

"This refinement process resulted in a final codebook consisting of eleven constructs."

Discussion¶

Related Claims¶

• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — a broader claim this one bears on
• Adding think-aloud data to theory prompts improves GPT-4o codebook completeness and alignment with the data — related
• GPT-4o prompted only with SRL theory references generates a codebook accurately representing most key elements of both foundational SRL theories — related
• A hybrid human-AI workflow using GPT-4o with retrieval-augmented generation supported efficient inductive thematic analysis while preserving researcher judgment — related
• GPT-5 used as a third coder introduced a calculation_check category absent from the human codebook, covering 28% of messages and exposing an unnamed recurring request type — related
• Inductive content analysis of 69 validated digital literacy frameworks yields eleven shared element categories spanning student and educator roles — related
• Fully inductive LLM codebook development risks importing unexamined sensitizing concepts, such as folk theories and scientific misconceptions, from the model's training data — related