Research Data Collection Methods

Element · All elements
Evidence · 4 claims (2 for, 1 mixed, 1 against) · 4 studies (2 causal, 2 review), q2 · 0 of 4 report an effect size · 4 claims rest on one study

Description

People tend to change their behavior when they know they're being watched — the Hawthorne effect. Observational research can be done naturalistically, in the participants' natural environment where they are usually unaware they're being observed (reducing the likelihood of the Hawthorne effect), or in a laboratory setting, where the researcher can control confounding factors and distractions at the cost of ecological validity and the expense of maintaining the controlled setting. A naturalistic classroom observation loses control — a substitute teacher on the observation day, for instance, could change student behavior and skew the data — that a laboratory setting would avoid, but at the price of studying behavior somewhat removed from its natural context.

Surveys trade off in a similar way depending on question format. Highly structured, forced-choice surveys (a fixed response set like "strongly disagree" through "strongly agree," or a numeric scale) let responses be coded and analyzed quickly with statistical software, but typically yield only surface information across many variables rather than deep understanding of any one. Open-ended surveys let each participant devise their own response, which can surface deeper insight but makes comparing answers across participants much harder. Because surveys rely on self-report, they are vulnerable to social desirability bias and related inaccuracies: participants may want to project a particular image, feel uncomfortable answering honestly, misjudge their own behavior, or simply lack awareness of the behavior being asked about.

Content analysis examines existing media — old texts, images, commercials, song lyrics — to explore patterns or themes in a culture (a classic example is Ariès's 1962 history of childhood, Centuries of Childhood, and content analyses of television commercials for sexual, violent, or ageist content are a similar case). This spares a researcher the time and expense of recruiting respondents, but limits conclusions to how accurately the analyzed media actually reflects the population's real actions and sentiments. Secondary analysis (archival research) goes further and analyzes data that has already been collected for some other purpose. It similarly avoids recruiting new subjects, but the researcher must assess the quality of the original data collection and is limited to whatever questions and data the original study happened to collect — a secondary analysis cannot ask a question the original data was never positioned to answer.

Design Implications

Context

Requirements

• A method chosen to match what the research question actually needs — naturalistic observation and open-ended surveys for depth and ecological validity, laboratory observation and forced-choice surveys for control and comparability

Constraints

• Any self-report method carries the risk that a participant's stated answer diverges from their actual belief or behavior, for reasons ranging from deliberate impression management to genuine lack of self-awareness
• Content analysis and secondary/archival analysis are both limited to the population and questions the original material or dataset can actually speak to, regardless of how interesting a new question might be

Target Learning Goals

• Choosing a data-collection method whose tradeoffs match the research question, rather than defaulting to whichever method is most familiar or convenient

Affordances

• Research Design Taxonomy — Quantitative/Qualitative and Descriptive/Correlational/Experimental

Claims

• End-of-course survey respondents strongly recommended the chatbot, but attitudinal outcomes showed no consistent treatment effects and survey participation was differentially selective [-W]
• Observation scores and student surveys were moderately correlated with student learning and likely captured different dimensions of teaching quality [+M]
• Most school-based intensive longitudinal studies collect data digitally (65.3%), and paper-and-pencil use has decreased significantly since 2020 [~W]
• Eye tracking can be used to study student engagement, teacher expertise, and teacher-student interaction in naturalistic classroom contexts [+W]

Related Elements

• Observation — the classroom-assessment use of observation is a related but distinct application from the research-methodology use described here
• Drawing laboratory with time-lapse photography of drawing evolution

Examples

• A researcher choosing naturalistic classroom observation over a lab setting specifically to avoid altering the behavior they want to study
• A study using archival school-district records instead of newly collected survey data, accepting the tradeoff of being limited to whatever the district happened to record

Key Sources

• Ariès, P. (1962). Centuries of childhood: A social history of family life. Vintage Books.
• Arduini-Van Hoose, N. (2020). Methods of data collection. In Educational psychology. Retrieved from https://edpsych.pressbooks.sunycreate.cloud. CC BY-NC-SA 4.0.

Page metadata

Field	Value
type	element
id	research-data-collection-methods
title	Research Data Collection Methods
description	Observation, surveys, and archival/content analysis are the main methods researchers use to gather data, each with a distinct tradeoff between ecological validity, depth, and cost that mirrors — but is methodologically distinct from — their use as classroom-assessment techniques.
status	review
canonical	True
generated	by claude/unspecified on 2026-08-29
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.