Skip to content

Standardized Test Fairness and Bias

Principle · All principles
Evidence · 48 claims (7 for, 38 mixed, 3 against) · 31 studies (11 associational, 7 causal, 7 review, 2 design, 2 theoretical, 1 quant-synthesis, 1 qualitative), q1–q3 · 4 of 31 report an effect size · 47 claims rest on one study

Conditional relationship

This page owns the fairness of a decision made from a standardized test score (an admissions or scholarship cut-off, a placement, a gifted or enrichment identification, a licensing or programme-entry test) across the groups of learners the decision is applied to. A score from a nationally or institutionally normed test is expected to support the same decision equally well for learners of different sex, race or ethnicity, social class, culture or home language only to the extent that three separate conditions hold for those groups and that use:

  • Item content and format do not depend on group background. No item is harder for one group than its overall scores predict for reasons unrelated to the construct (a word sense, a context, the people pictured), and the response format does not favour one group's strengths over the capability being measured.
  • The score predicts the criterion equally well for each group. Where the score is used to forecast later performance (first-year grades, course success), it predicts with similar accuracy and without systematic over- or under-prediction for each group.
  • The testing conditions do not change performance by group. High-stakes framing, identity reminders before the test (a demographic questionnaire), time pressure and anxiety do not lower one group's scores below what the same learners show under other conditions.

The outcome is assessment-accuracy: whether the decision made from the score is the one each learner's capability and later performance warrant. It is not a learning outcome. The three conditions are independent: a test can pass an item-bias review and still predict differently by group, and a test can predict equally well and still be taken under conditions that depress one group's scores.

No claim in the wiki tests the relationship as a whole. None compares decisions made from standardized scores with and without item-bias screening, local prediction checks or changed testing conditions, and the wiki holds no claim page on stereotype threat itself. The claims that bear on it are of three kinds: congressional testimony from 1987 reporting College Board and ETS figures second-hand (that the SAT under-predicts girls' grades, that items with male figures favour males, that score cut-offs gave boys most places and awards), narrative reviews attributing the under-identification of culturally and linguistically diverse students for gifted programmes to biased measures, and psychometric studies showing that methods for flagging biased items disagree and that an item's position changes its difficulty. Two randomized field experiments and a large replication test a values-affirmation exercise, the one intervention here aimed at the testing-conditions mechanism, and they disagree. Everything below is a default design built from the earlier page's three mechanisms, with each step's evidence status stated.

How the converted sibling sits beside this: Validity, Reliability, and Bias in Classroom Assessment owns the quality of the inference a teacher draws from a classroom assessment, including content-bias review of teacher-made items and item analysis after a classroom test; it does not repeat here. This page holds what is specific to standardized tests used for selection or placement across groups: differential prediction, statistical item-bias screening on large samples, and the testing conditions of a high-stakes sitting.

Default design, while the relationship is untested

The earlier page's three mechanisms, turned into a procedure for someone who uses standardized scores to make decisions about learners, prepares learners to sit such a test, or writes items for one. It runs over a decision cycle (preparation, the sitting, the decision, and a check once the outcomes are known), not one lesson. Each step's evidence status is stated beside it.

  1. Name the decision, the groups and the criterion before using a score. Write down what the score will decide (admission, a scholarship, placement in a set, entry to a gifted programme), for which groups you need the decision to be equally accurate, and what later outcome the score is supposed to predict. Untested; from the earlier page's point that "is this test biased?" is three questions, not one.
  2. Do not let a single score cut-off decide entry. Combine the score with at least one other piece of evidence the decision rests on (school grades, a work sample, a performance task), and look again at every learner within a band of the cut-off rather than using the cut as a wall. As a proposal, the band is the test's reported standard error of measurement either side of the cut. Basis: testimony reports that girls, who earned higher high-school and college grades, averaged lower SAT scores than boys (SAT scores underpredict girls' performance [~W]), that MIT admitted women with lower SAT mathematics scores who did as well as men in freshman mathematics (MIT women with lower SAT math scores performed as well [~W]), and that cut-off-based selection gave boys most talent-search invitations and 64 per cent of National Merit scholarships (test scores gate gifted programme entry [~W]; National Merit awards skew male [~W]). All second-hand testimony from 1987, with no effect size; none tests a combined-evidence rule.
  3. Check differential prediction locally, once the criterion is known. When the first-year grades (or other criterion) for an admitted cohort are in, regress the criterion on the score separately for each group, or with a group term, and look at both accuracy (how well it predicts) and direction (whether a group's actual outcomes run above or below its predicted ones). Do not assume the direction: the earlier page reports Young (2004) finding lower prediction accuracy for Black, Latino and female students, with scores slightly over-predicting Black and Latino students' grades and under-predicting women's. If a group's outcomes are under-predicted, a common cut-off excludes some of its learners who would succeed; weight the other evidence more for that decision, or set the cut from the local regression. Untested in the wiki as a procedure; the under-prediction of girls is the testimony above ([~W]).
  4. If you write or select items, screen them in two ways, and treat a statistical flag as a question. Have reviewers from varied backgrounds read every item for word senses, contexts and people that one group is more likely to know or see (the earlier page's "field" example, from Popham 2004), and balance the people shown. Then, with large samples, run a differential item functioning (DIF) analysis with more than one method, and send every flagged item to a content review before removing it. Basis: testimony citing a 1979 ETS study that females did better on items with female or neutral figures than with male figures, and a count of 42 men to three women in 24 SAT reading passages (male-oriented item content [~W], second-hand); two DIF methods applied to the same state science performance assessment agreed on only three of seven flagged items each (ICC and LDF methods identify different DIF items [~M]); on a physics test, response-time differences between boys and girls did not anticipate which four items were flagged (DIF not predictable from response times [~M]). The earlier page adds that standard review did not remove the small verbal-SAT pattern Freedle (2002) reported, in which white test-takers did relatively better on easy items and African American, Hispanic American and Asian American test-takers on hard ones.
  5. Keep item positions fixed across forms, or check them. If several booklets or shuffled forms are used, keep each item in the same position, or compare its difficulty across positions before treating forms as equivalent; a position effect can look like group bias when groups take different forms. Basis: on a Turkish national examination, an item placed later in one booklet was harder for that booklet's takers, which the authors attribute to fatigue (later-positioned item more difficult [~M]), and items flagged for DIF generally sat at very different positions across booklets (DIF items differently positioned [~M]).
  6. Offer more than one response format where the construct allows it. If the capability can be shown in writing as well as by choosing an option, include both, and report them separately. Untested here; from the earlier page (Hoover 2006: adding an essay to the SAT raised women's verbal scores relative to men's). Related argument: a test in a mainly verbal code may understate what some learners know (verbal tests may underestimate knowledge [~W], an argument, not a result).
  7. Remove identity reminders and needless high-stakes framing from the sitting. Put demographic questions after the test, not before it; describe the test by what it is for ("this shows what to work on next", where that is true) rather than as a measure of intelligence or ability; give the same neutral instructions to every group. Untested in the wiki: the earlier page's account of stereotype threat (Aronson & Steele, 2005) is not on any claim page here. Removing a demographic form from before the test costs nothing, which is why it is the default.
  8. Rehearse the format at low stakes before the real sitting. Give learners two or three practice sittings in the test's own format and timing, scored with feedback and not counted. Basis, carried from neighbouring settings: mathematics anxiety correlates r = −.28 with mathematics achievement across 223 studies (math anxiety degrades performance [~M], correlational, does not test practice sittings); in team-based learning, dental students in a revised low-stress format scored 5.2 per cent higher on individual assessments than medical students in a high-stakes format (low-stress TBL, higher iSAT scores [~M], two cohorts, not randomized), and the high-stakes group rated the questions less fair (high-stakes TBL questions seen as less fair [~M], survey).
  9. Use a values-affirmation exercise only as a trial, and measure it. A 15-minute writing task about a personally important value, early in a term or before a high-stakes sitting, raised African American seventh-graders' course grades in two small randomized field experiments, but a large replication with the same protocol in the same district found no effect (d = −0.07) (self-affirmation: raised grades, then a null replication [~M]). It was tested on term grades, not on standardized test scores. If you use it, compare affirmed and control classes on the outcome you care about before adopting it.
  10. Report scores and decisions by group, with their uncertainty. For each decision cycle, report the share of each group selected, the score distributions, and what the local prediction check found. Untested proposal; the earlier testimony's own figures (scholarship and invitation shares) are the kind of report that made the pattern visible.

Across a decision cycle (dose and progression; all quantities are untested proposals). Preparation, over the four to six weeks before a high-stakes sitting: an untimed practice sitting in the first week with feedback on every item, a timed one at mid-point, a full-length timed one in the last fortnight, each 45 to 90 minutes depending on the real test's length; move a learner from untimed to timed practice once they finish the untimed version with time to spare. The sitting: demographics after the test, neutral instructions. The decision: second evidence for every learner in the band around the cut. After one year, when outcomes are in: the differential-prediction check (step 3) and, for item writers, DIF on the administered items (step 4); after three cycles, revise the cut-off or the weight given to the score from the accumulated local data. A new test, a new population or a new use restarts the cycle at step 1.

When a condition is not met: if groups are too small for a regression or a DIF analysis (as a proposal, fewer than about 100 learners per group), rely on the content review and on the second evidence near the cut, and pool cohorts across years before running the statistics; if you cannot change the test or its sitting (a national examination), the steps still open to you are 2, 3, 8 and 10.

Fitting the design to a situation

The three facts that most change the decision:

  • What the score decides, and whether you control the test. A user of an external test (an admissions office, a gifted programme) can change how the score is combined and where the cut sits, not the items; an item writer or test owner can change the items and the sitting; a teacher preparing learners can change only the preparation. If unstated, ask: what does the score decide, for whom, and which of the test, the sitting and the decision rule can you change?
  • Which groups the decision must be fair to, and how many learners are in each. Prediction checks and DIF statistics need sizeable groups; small groups leave you with content review and second evidence. If unstated, ask: which groups are in the population (sex, race or ethnicity, home language, social class, disability), and roughly how many test-takers in each per cycle?
  • Whether a later outcome is available to check the score against. Differential prediction can only be checked when the criterion (grades, completion, job performance) is recorded for the people the score selected. If unstated, ask: what later outcome is the score supposed to predict, and is it recorded for each group?
If the brief says… Then change… Basis
The score gates admission or a scholarship with a single cut-off Add a second piece of evidence (grades, a work sample) for every applicant within a band of the cut; run a differential-prediction check once first-year outcomes are in, and do not assume which direction any gap runs SAT underpredicts girls' performance [~W]; National Merit awards skew male [~W] (both 1987 testimony, second-hand); the directions by group are the earlier page's (Young 2004)
The decision is gifted or enrichment identification Screen every learner rather than relying on teacher referral; use more than one measure, including a performance or classroom-learning measure; compare each group's identification rate with its share of enrolment every year Culturally and linguistically diverse students under-identified nationwide [~W] and teachers under-refer them [~W] (one issue brief); a narrative review attributes under-representation to biased measures [~W] (second-hand, Ford 2003); traditional identification tests omit classroom-learning components [~W] (argument). None tests universal screening or multiple measures
Learners belong to a group negatively stereotyped in the tested domain (women in mathematics; Black or Latino students in academic tests) and the sitting is high-stakes Move demographic questions to after the test; describe the test by its use, not as a measure of ability; consider a values-affirmation exercise only as a measured trial Untested in the wiki (stereotype threat has no claim page); affirmation: raised grades in two small trials, null in a large replication [~M] (seventh graders, term grades, not test scores)
Learners are anxious about tests, or the test is timed and speed is not the construct Two or three untimed, then timed, practice sittings in the real format with feedback; ask the test owner whether extended time is allowed; read a single timed high-stakes score with caution Math anxiety degrades performance [~M] (meta-analysis of correlations; does not test these moves); higher anxiety goes with smaller testing effects [~M] (retrieval practice, review chapter)
Learners are taking the test in a second language, or have low literacy Use the accommodations the test allows (glossaries, read-aloud for non-reading tests, extra time); do not read a low score as low subject knowledge without a second, low-language measure; for items you write, keep reading load below the construct Verbal tests may underestimate subject knowledge [~W] (argument); minority-testing research has rarely examined alternative symbolic forms [~W] (a 1986 review noting the gap)
Learners are young children (child) taking a screening or readiness test Do not use one sitting for a placement; observe over several occasions; use pictures and spoken instructions; re-screen within the year untested proposal
You write or own the items (a district, an awarding body, an assessment product) Content review by reviewers from varied backgrounds, then DIF with at least two methods on large samples; every flagged item gets a content review before it is dropped; record which method flagged it ICC and LDF methods flag different items [~M]; item purification flags more items [~M]; NC-DIF chi-square over-sensitive at large n [~M] (simulation, attributed to Fleer 1993); male-oriented item content [~W]
Groups are small (a single school or programme, under about 100 per group) Skip DIF and regression statistics for now; rely on content review and on second evidence near the cut; pool several years before running them untested proposal; DIF Type I error rates rose with more and larger DIF items in simulation (DIF Type I error increases with DIF items and magnitude [~M]), so small or contaminated samples give unstable flags
Several forms or booklets are used Keep item positions fixed across forms, or check each item's difficulty by position before comparing groups who took different forms Later-positioned item more difficult [~M]; DIF items differently positioned across booklets [~M] (one national examination)
The test is taken in mixed modes: some in a centre, some remote, some on phones only Give the same items, time and aids, or record the difference beside each score; avoid long passages and wide tables on phones; compare score distributions by mode before pooling, and check that mode is not standing in for a group (rural, low-income) untested proposal
Adults in work settings: a licensing, promotion or job-entry test Check that the items sample the job's tasks, not general reading or cultural knowledge; validate the score against observed job performance by group; add a work-sample task for candidates near the cut untested proposal; the prediction check (step 3) applied to a job criterion
One teacher or tutor prepares many learners, or facilitator time is scarce (volunteers, a short weekly session) Use one full practice sitting under real conditions, self-marked against an answer key with worked reasons, and spend the scarce contact time only on the three items most learners missed; move demographic forms after the test, which needs no time untested proposal
Only a single session is available before the real test Use it for one timed practice in the real format with immediate feedback and a walk-through of the instructions; do not add new content untested proposal
The brief names its own routine (its own admissions rubric, test-prep programme, identification checklist) Keep it; change only what the evidence here says matters: no single-score cut without a second look near it, demographics after the test, and a by-group check of who is selected and how they later do untested; the three changes follow the three mechanisms above
Results are reported publicly or drive accountability (school league tables, programme funding) Report by group with the number of test-takers and the uncertainty; do not compare groups on a score whose prediction has not been checked for them; link the use to High-Stakes Testing and Accountability Effects untested proposal

Eight of the fifteen rows cite a claim; none of those claims tests the move in its row. They come from congressional testimony reporting figures second-hand, an issue brief and narrative reviews on gifted identification, psychometric studies of one examination or simulations, a correlational meta-analysis of anxiety, and field experiments on term grades, and each is carried to the decision in its row. The small-groups row is an untested proposal that cites a claim only for its limit; the young-children, mixed-mode, work-setting, scarce-time, single-session, named-routine and public-reporting rows are untested proposals.

Observation, state and explanation

What can be observed is a score, an item response, a selection decision and, later, an outcome, each for a learner whose group is recorded. "The test is biased against this group" or "this learner is under threat" are inferences. A group difference in scores is not by itself evidence of bias, and an absence of difference is not evidence of fairness.

Response Plausible explanations Observation that could change the interpretation
One group does worse on a few items than its total scores predict The items draw on background or word senses one group meets less; the items sit at a different position in that group's form; a chance flag from one method Read the items for outside knowledge and the people shown; re-run with a second DIF method; compare the item's difficulty at the same position; see whether the flag recurs next cycle.
A group's later grades run above what its scores predicted (under-prediction) The test misses capabilities that matter for the criterion; the group's scores were depressed by testing conditions; the criterion is graded differently for that group Compare with a second measure taken under low-stakes conditions; check whether the gap holds across courses and graders.
A group's later grades run below what its scores predicted (over-prediction) Conditions after admission (support, belonging, finances) differ by group; the criterion is graded harder for that group; the test measures something the group had more coaching on Look at completion and support use by group; compare grades from different instructors. The earlier page notes no settled explanation for either direction.
A learner scores much lower on the real sitting than on practice sittings Anxiety or high-stakes pressure; an identity reminder at the sitting; a different form or harder positions; ordinary variation Ask the learner what got in the way; check what preceded the test (forms, instructions); compare with a re-sit under the same conditions as practice.
One group is selected far less often than its share of the eligible population The score differs by group for reasons unrelated to the criterion; referral rather than the score kept the group out; real differences in prior opportunity Check who was referred or invited to test, not only who passed; run the prediction check; compare with a second measure.
Women score lower on a multiple-choice section but not on an essay section of the same domain A format effect; the sections sample different content Compare items covering the same content in both formats.

These rows are proposals, untested as diagnostics. Each probe is a further measurement with its own error, so record it and keep the explanations that remain open rather than choosing the one whose name fits.

Evidence and qualifications

No claim in the wiki tests whether decisions from standardized scores become fairer across groups when these moves are made. The claims below bear on parts of it.

The claims differ in learners, instruments and settings (1980s SAT figures, gifted identification, a state science assessment, a physics test, seventh-graders' grades, anxiety studies), so do not rank them by effect label. The testimony claims carry figures that are almost forty years old and second-hand. None measures whether a selection decision became more accurate for any group.

The evidence by question

Further evidence, not yet read against this model

Claims found while rewriting, used in the default design or the situation table for what they are, or neighbouring the page. None tests the relationship.

Objective and learner-valued goal

The designer's or institution's objective is a selection or placement decision that is equally accurate for every group it is applied to. Learners value something related but different: a place, a scholarship, a licence, being seen as able, or simply not being judged by a number they think misses what they can do. Ask them, before the sitting, what the result will open for them and what they expect it to show; ask afterwards whether it showed what they can do and, if not, what got in the way. A learner who says "I froze once I'd filled in the form" or "I didn't know what the word meant there" is reporting a possible construct-irrelevant demand; record it as an observation, not a complaint.

For example, a first-generation applicant to an engineering programme may want admission and the scholarship that makes it affordable, while the admissions office's objective is predicting first-year mathematics grades. A score near the cut supports the office's prediction only as far as the score predicts equally well for applicants like her; her goal needs the second look near the cut and, a year later, the prediction check that shows whether students like her were being under-predicted.

What would revise this model?

The expectation should weaken if local differential-prediction checks across groups, done with current tests and adequate samples, show accurate and unbiased prediction for each group, in which case the second look near the cut becomes a reliability measure, not a fairness one. If content review and DIF screening do not change group differences on the items that remain, the review stays as a duty to avoid offence, not an accuracy claim. If moving demographic questions after the test and changing the framing make no difference to any group's scores in controlled comparisons, step 7 should be dropped as a fairness move, though it costs nothing. If values-affirmation fails again in replications, step 9 should go. Do not protect the model by calling every group gap "bias" or every absence of a gap "fairness" after the fact.

Item bias, differential prediction and the effect of testing conditions are separate claims, each needing its own evidence, and the wiki holds only second-hand or neighbouring evidence for each. The present evidence establishes no acceptable size of prediction gap, no DIF method to trust over another, and no intervention at the sitting that reliably changes a group's scores.

Claims

Examples

Key Sources

  • Popham, W. J. (2004). America's "failing" schools: How parents and teachers can cope with No Child Left Behind. RoutledgeFalmer.

  • Freedle, R. O. (2003). Correcting the SAT's ethnic and social-class bias: A method for reestimating SAT scores. Harvard Educational Review, 73(1), 1-43. doi:10.17763/haer.73.1.8465k88616hn4757

  • Hoover, E. (2006). Panel says gender gap in SAT scores may be linked to writing section. The Chronicle of Higher Education.

  • Young, J. W. (2004). Differential validity and prediction: Race and sex differences in college admissions testing. In R. Zwick (Ed.), Rethinking the SAT: The future of standardized testing in university admissions (pp. 289-301). RoutledgeFalmer.
  • Aronson, J., & Steele, C. M. (2005). Stereotypes and the fragility of academic competence, motivation, and self-concept. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 436-456). Guilford Press.
  • Arduini-Van Hoose, N. (2020). Issues with standardized tests. In Educational psychology. Retrieved from https://edpsych.pressbooks.sunycreate.cloud. CC BY-NC-SA 4.0.
  • Sex and Race Differences on Standardized Tests: Oversight Hearings before the Subcommittee on Civil and Constitutional Rights of the Committee on the Judiciary, House of Representatives, One Hundredth Congress, First Session. (1989). https://eric.ed.gov/?id=ED312276
  • Soysal, S., & Yilmaz Kogar, E. (2021). An Investigation of Item Position Effects by Means of IRT-Based Differential Item Functioning Methods. International Journal of Assessment Tools in Education, 8(2), 239-256. https://doi.org/10.21449/ijate.779963
  • Gifted and talented students at risk for underachievement. (2008). Center for Comprehensive School Reform and Improvement Issue Brief. https://www.centerforcsri.org
  • Shavelson, Richard J.; Webb, Noreen M.; Lehman, Penny. (1986). The Role of Symbol Systems in Problem Solving: A Literature Review. CSE Report No. 269. https://eric.ed.gov/?id=ED338680
Page metadata
Field Value
type principle
id standardized-test-fairness-and-bias
aliases similar-item-positions-across-booklets, multiple-criteria-gifted-identification, tests-capitalizing-on-cultural-strengths
title Standardized Test Fairness and Bias
description A standardized test score is expected to support a selection or placement decision equally well across groups only where its items do not depend on group background, it predicts the later outcome equally accurately for each group, and the testing conditions do not depress one group's scores; no claim tests this relationship, and the claims bearing on it are second-hand testimony, narrative reviews, psychometric studies of item-bias methods and a values-affirmation intervention that did not replicate.
status review
canonical True
generated by claude/unspecified on 2026-10-05
trust unverified — no one has confirmed this page's content
sources 3 — soysal-2021, gifted-and-talented-students-at-risk-for-underachievement-2008, shavelson-1986

Frontmatter is the page's machine-readable record — see the Schema & Guide.