Claims

Independent data-analysis teams given the same data and question produced widely varying answers, illustrating analytic flexibility

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims

Reading the evidence codes

q is the design tier, i the impact, n the sample, and the kind with its r the rigour judged within that kind: what each code means.

This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.

q2 i? In the red card study, 29 teams of analysts answering the same research question with the same dataset produced 29 different answers, ranging from no difference to nearly three times more likely. → Mellor 2021

Evidence

Mellor 2021

Mellor, D. (2021). Improving norms in research culture to incentivize transparency and rigor. Educational Psychologist, 56(2). https://orcid.org/0000-0002-3125-5888

q2 · i? · theoretical · r3

The article reports, citing Silberzahn et al. (2018), that 29 teams of data analysts given the same dataset on soccer referees and red cards produced "29 different answers", with statistical significance and uncertainty varying widely. No effect size is printed.

"With the same research question and the same data, one might expect that there would be one answer, perhaps with a reasonable band of uncertainty. There were, however, 29 different answers."

Discussion

Page metadata
Field Value
type claim
id many-analysts-one-dataset-variable-answers
title Independent data-analysis teams given the same data and question produced widely varying answers, illustrating analytic flexibility
description Independent data-analysis teams given the same data and question produced widely varying answers, illustrating analytic flexibility
status review
evidence_strength weak
newest_evidence 2021
generated by process:wiki-ingest on 2026-10-22
trust unverified — no one has confirmed this page's content
sources 1 — mellor-2021

Frontmatter is the page's machine-readable record — see the Schema & Guide.