Codebook development, theme generation, and community validation of findings¶

Strategy · All strategies
Evidence · 1 claim (1 mixed) · 1 study (1 qualitative), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description¶

Phase 3 analysis guidance has teams familiarize themselves with transcripts, create an initial codebook documenting 'each code's meaning and application', and code with an Excel template or qualitative software, revising codes as new ones emerge. Themes are generated by grouping codes, refining and interrogating them, defining each theme, and writing up findings with illustrative quotes. Step 5 states: 'Always share findings with the community representatives who contributed' so partners 'feel their contributions are truly represented.'

Design Implications¶

Context¶

Requirements¶

• Check with other coders and relevant stakeholders to confirm themes are accurate, complete, and culturally informed

Constraints¶

Target Learners¶

• Agency staff analyzing qualitative community engagement data

Target Learning Goals¶

• Generate, validate, and report trustworthy themes from engagement data

Affordances¶

• Three Phase Community Engagement Process

Claims¶

• GPT-5 used as a third coder introduced a calculation_check category absent from the human codebook, covering 28% of messages and exposing an unnamed recurring request type [~W]

Related Strategies¶

• Collective Data Analysis
• Plan engagement by defining objectives, aligned questions, engagement type, and piloted materials

Examples¶

-

Key Sources¶

• Collecting and Making Sense of Community Engagement Data. (2025). Regional Educational Laboratory Northwest. https://ies.ed.gov/rel-central/2025/01/dissemination-plan-template