Collaborative Critique and Redesign of Flawed Studies¶

Strategy · All strategies
Evidence · no claims cited

Description¶

Requiring students to redesign a flawed study, not just critique it, forces them to grapple with the epistemic ideals that underlie design choices (validity, objectivity, adequate sampling) rather than mechanically applying a checklist. The strategy pairs three ingredients: an authentic popular-media source reporting a psychological finding (so students encounter evidence as it actually circulates, not as a sanitized textbook example), a phenomenon that is intentionally difficult to operationalize (e.g., emotions, therapeutic "depth"), and a small-group discourse requirement that forces students to justify design choices to peers rather than submit an individual answer.

Design Implications¶

Context¶

Requirements¶

• A popular-media source (newspaper article, talk, podcast) reporting a psychological claim with identifiable methodological weaknesses
• A synchronous small-group format (the source study used groups of five, 90 minutes, producing one collaborative response) so that design choices must be argued for, not just individually assumed
• A phenomenon or construct that lacks an obvious, ready-made operational definition, so students must work through construct-validity tensions rather than look one up

Constraints¶

• Without pairing critique with an actual redesign task, students tend to stop at surface-level criticism (e.g., "small sample") without engaging the underlying ideal (why does sample size matter for this claim specifically)
• A single collaborative task per topic does not, by itself, teach students to evaluate the credibility of the source itself (author expertise, outlet credibility) — that requires a separate emphasis if it is also a goal

Target Learners¶

• Introductory-level students encountering psychological/scientific evidence evaluation for the first time
• Grasp of Evidence framework — extends this general framework to evidence evaluation in the social/behavioral sciences specifically

Target Learning Goals¶

• Shifting evidence evaluation from ideals grounded in personal experience/intuition toward disciplinary ideals (valid methods, objectivity-as-procedure, replication)
• Distinguishing a phenomenon's conceptual definition from its operational (measurable) definition

Instructions¶

1. Present a popular-media report of a psychological finding with real, identifiable design flaws (e.g., an underpowered or poorly controlled study, an ill-defined construct).
2. In small groups (synchronous, ~90 minutes), have students identify the study's methodological problems.
3. Require the group to produce an alternative study design addressing those problems — including an explicit operational definition of the key construct.
4. Require a short written justification of key design choices (why this sample, why this measure) so the group must externalize the epistemic ideal behind each choice.
5. Repeat across 2-3 tasks spanning different phenomena over a course, so students encounter the same ideals (validity, objectivity, sampling) applied to different construct-definition challenges.

Related Strategies¶

• (none yet linked)

Examples¶

• Groups evaluated the "Little Albert" experiment's ethical and methodological problems, then designed a replication with a proper sample, consent procedures, and controls.
• Groups evaluated an emotion-detection algorithm's construct validity (was it actually measuring emotion, or self-reported perception of emotion?) and designed an alternative approach combining multiple measurement instruments.
• Groups operationalized "depth" in psychotherapy — initially treating it as an unmeasurable, purely subjective quality, then converging on a specific, defensible operational definition (e.g., "focuses on a problem's underlying sources") sufficient to design a comparative study.

Key Sources¶

• Dishon, G., Barzilai, S., & Yanai, J. V. (2024). Grasping Psychological Evidence: Integrating Evidentiary Practices in Psychology Instruction. Cognition and Instruction, 42(1), 56-91. https://doi.org/10.1080/07370008.2023.2248641