Collaborative critique-and-redesign instruction shifts students from a value-free view of objectivity toward a procedural one¶

Claim · All claims
Evidence · 1 study · design r2 · q3 · i? · n=120 first-year undergraduates (24 groups of 5), one Introduction to Psychology course, Israeli research university
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

After a semester of collaboratively critiquing and redesigning flawed psychological studies, introductory psychology students significantly increased their use of disciplinary evidence-evaluation ideals (valid methods, replication, procedural objectivity) and significantly decreased reliance on personal experience and a naive, value-free notion of objectivity — while their evaluation of lay evidence sources (source credibility, outlet reliability) did not significantly change.

Subclaims¶

q3 i? The ideal of "valid methods" (whether a study actually measured what it claimed to) rose from 2% to 38% of students' pre/post written evidence evaluations (McNemar's test, p<.001) — a roughly 19-fold increase. → Dishon et al. 2024

q3 i? Naive appeal to "objectivity" as researcher freedom from bias fell from 41% to 17% of responses (p<.001), while explicit appeal to replication rose from 24% to 55% (p<.001) — consistent with a shift from viewing objectivity as an individual, value-free stance toward viewing it as an emergent property of reliable, replicable procedure. → Dishon et al. 2024

q3 i? Reliance on personal coherence (agreement with one's own experience/intuition as a basis for judging a claim) fell from 69% to 52% of responses (p=.005), though about half of students continued to invoke it post-instruction — suggesting students learned to coordinate personal and disciplinary knowledge rather than abandon the former. → Dishon et al. 2024

q3 i? None of the four ideals or processes specific to lay evaluation of evidence (source expertise, outlet credibility, source benevolence/integrity) showed significant pre/post change (p=.061-.503), indicating the course's emphasis on disciplinary evidence evaluation did not generalize to evaluating the credibility of a source itself. → Dishon et al. 2024

Evidence¶

Dishon et al. 2024¶

Dishon, G., Barzilai, S., & Yanai, J. V. (2024). Grasping Psychological Evidence: Integrating Evidentiary Practices in Psychology Instruction. Cognition and Instruction, 42(1), 56-91. https://doi.org/10.1080/07370008.2023.2248641

q3 · peer-reviewed single-group pre/post design with validated coding scheme (Grasp of Evidence and AIR frameworks), high interrater reliability (Cohen's κ M=0.90, SD=0.07), and McNemar's tests appropriate to matched binary pre/post data; no control group (authors cite ethical reasons for not withholding instruction) i? · no standardized effect size printed; statistically significant shifts in response proportions on several disciplinary ideals (e.g., 2%→38% on valid methods, McNemar's tests), and the design cannot isolate which specific course elements caused the shift n=120 first-year undergraduates (24 groups of 5), one Introduction to Psychology course, Israeli research university · design · r2

First-year psychology students completed parallel written evidence-evaluation tasks (evaluating popular reports of psychological findings) before and after a 12-week course built around Collaborative Critique and Redesign of Flawed Studies — three synchronous small-group tasks requiring critique and redesign of flawed studies on child-rearing, emotion measurement, and psychotherapy. Coded against the Grasp of Evidence framework (Duncan et al.) and the AIR model of epistemic cognition (Chinn et al.), responses showed large, statistically significant shifts toward disciplinary ideals like valid methods and procedural objectivity, alongside a smaller decline in reliance on personal experience — but no significant change in students' evaluation of lay evidence sources (credibility, expertise), which the course did not explicitly emphasize. This extends Epistemic Cognition to evidence evaluation in the social/behavioral sciences specifically.

Discussion¶

The absence of a control group is a real limitation the authors acknowledge (for ethical reasons, they did not want to withhold the intervention from any students); the pre/post design cannot rule out maturation, other coursework, or general university exposure as contributing causes. The qualitative discourse evidence supporting the mechanism (why collaborative redesign works) comes from an in-depth analysis of a single group across all three tasks and should be read as illustrative rather than a separately quantified effect. The honest null result on lay evidence evaluation is itself informative: it shows the shift was not a general increase in critical thinking but was specific to the disciplinary ideals the course actually emphasized.

Learner Variables¶

• Epistemic Beliefs — outcome: instruction changes it

Related Claims¶

• Both instructional conditions produced statistically significant pre-test to post-test gains in NOS and SK views, with larger effect sizes in the experimental group — the general claim this finding bears on, with the studies behind it

• Civic Online Reasoning Instruction Improves Evaluation — the general claim this finding bears on, with the studies behind it

• (none yet linked)
• Course-based emphasis on reframing and stakeholder inquiry shifts pre-service teachers toward informed-designer stances — related
• After an early algebra intervention, the number of kindergarteners giving a relational definition of the equal sign went from one to nine, and acceptance of non-standard equation forms rose, as reported from the authors' earlier study — related
• Students' experiment-evaluation strategies shifted toward correct CVS after instruction, though 23% still used incorrect strategies — related
• Students in a collaborative, project-based engineering design course gained declarative knowledge but not procedural or principled knowledge or personal interest — related
• A student-built CTE webpage redesign coincided with the school adding one computer science course the following year — related