Use practice rounds with independent dual extraction and third-reviewer consensus to strengthen data extraction consistency¶

Strategy · All strategies
Evidence · no claims cited

Description¶

To strengthen consensus in data extraction, the team "first conducted practice rounds in which multiple researchers independently extracted data from a set of articles, followed by structured discussions to reconcile discrepancies and agree on convention for recording the extracted data." In the formal process each article was coded independently by two researchers, with a third reviewer of relevant content expertise resolving unresolvable discrepancies. Structured tools such as multiple-choice or checklist-based coding items reduced subjectivity and enhanced interrater consistency.

Design Implications¶

Context¶

Requirements¶

• Structured coding formats (multiple-choice or checklist items) where possible; clear specification of source sections, page numbers, and whether entries are exact quotations or summaries

Constraints¶

• Open-ended items require careful attention to minimize variation in interpretation; concise summaries may lower interrater reliability when exact wording is required

Target Learners¶

• novice researchers learning data extraction and coding

Target Learning Goals¶

• interrater consistency in coding
• familiarity with research reporting conventions

Affordances¶

• Sr As Intentional Learning Experience Mentoring

Related Strategies¶

• Adopt a conservative abstract-screening strategy with a 'maybe' category and keyword exclusion filters to prevent premature exclusion

Examples¶

-

Key Sources¶

• Xiyu Wang, Fatemeh Dadashipour, Basori, Yukiko Maeda, & Jennifer C. Richardson. (2026). Scaffolding systematic reviews in learning design and technology through mentoring and AI integration. Educational technology research and development. https://doi.org/10.1007/s11423-026-10629-8