Skip to content

Validity, Reliability, and Bias in Classroom Assessment

Principle · All principles
Evidence · 265 claims (113 for, 134 mixed, 18 against) · 195 studies (73 associational, 46 design, 28 review, 22 theoretical, 17 causal, 6 quant-synthesis, 3 qualitative), q1–q3 · 17 of 195 report an effect size · 258 claims rest on one study

Conditional relationship

This page owns the quality of the inference a teacher or designer draws from a classroom assessment score, whatever the format. A score on a quiz, test, performance task or rating is expected to support a decision about a learner (reteach, regroup, grade, place, certify) only to the extent that three distinct conditions hold for that learner and that use:

  • Validity of the interpretation. The tasks sample the intended capability in the intended proportions (content), performance depends on that capability and not on something entangled with it, such as reading load, English proficiency, speed or anxiety (construct), and, where the score is used to predict, it does predict the later outcome (criterion). Validity belongs to an interpretation and a use, not to the test: the same fractions test can support a conclusion about a fluent-English learner's understanding of fractions and not about a recent immigrant's.
  • Reliability. The score would be similar on another occasion, with other items from the same domain, and from another rater. It rises with more items or tasks, clearer wording and clearer scoring criteria. A reliable score can still be invalid; an invalid one gains nothing from being consistent.
  • Freedom from bias. Neither the content nor its administration changes measured performance because of group membership: no offensive stereotypes, and no item that rewards background experience some learners had outside class and others did not.

The outcome this relationship is about is assessment-accuracy: whether the decision made from the score is the one the learner's actual capability warrants. It is not a learning outcome. Learning follows only if a more accurate decision leads to a better next step, which is the subject of the converted pages beside this one.

No claim in the wiki tests the relationship as a whole: none compares decisions made from classroom assessments built with a table of specifications, more items, calibrated scoring and a bias review against decisions made without them, or measures whether those moves change what happens to learners. The claims below test parts of it in other settings: that raters differ in severity, that rubric guidance improved peer raters' placement of writing samples on complex criteria, that ignoring passage-based item dependence overstates reliability, that item position changes difficulty, that bias-detection methods disagree, that a pass/fail decision is least consistent where scores bunch, that anxiety is associated with lower mathematics scores, that rapid guessing and low effort distort individual scores (and that alerting a proctor to it was followed by higher performance and better convergent validity), that AI and LLM scorers agree with teachers only moderately and least well on mid-range and linguistically weak responses, and that high school grades predicted later attainment better than single test scores did, that state-test word problems were disproportionately hard for English-language learners because of their language, that peer marks tracked a teacher's on exams with a clear marking scheme, that grades and test scores differ by group in ways neither measure alone accounts for, and that teachers' judgements of their students' performance ran high. Most come from large-scale testing, research instruments or single classes, and are carried to classroom assessment by extrapolation.

How the converted siblings sit beside this: Competency-Based Assessment applies these properties to one kind of judgement, a criterion-referenced "demonstrated / not yet" decision against a threshold, and owns its threshold-setting and assessor-agreement model; this page holds the general measurement properties any classroom score needs and does not repeat that model. Assessment for Learning owns how evidence is used during instruction; it presupposes evidence trustworthy enough to act on, which is this page's subject. The Constructive Alignment pattern owns whether outcome, activities and assessment call for the same capability at the same level, which is the content-validity question at course scale; the Authentic Assessment pattern is one design that raises construct match for performance goals at a cost to scoring consistency. Standardized Test Fairness and Bias holds large-scale cases.

Default design, while the relationship is untested

The earlier page's guidance, kept as a concrete default a teacher or designer can act on and revise. It is a procedure for building and using an assessment, run over a term, not a lesson. Each step's evidence status is stated beside it.

  1. Name the decision before writing an item. Write down what the score will be used for (decide whom to reteach tomorrow, award a unit grade, place in a set, predict who needs help before a spring test) and for whom. Every later check is made against that use. Untested; from the earlier page's definition of validity as the justification of an interpretation and use, and close to one article's argument that a classroom assessment should be judged by whether it serves the decision its user has to make (shift from validity to effectiveness [~W], argument, no data).
  2. Plan the sample with a table of specifications. Cross content areas against objectives, and set a planned item count in each cell before writing, in proportion to teaching time and importance, so over- and under-sampled content shows in advance. As a proposal, no objective that the decision rests on gets fewer than three items or one full performance task. Untested; from the earlier page (Linn & Miller).
  3. Ask for the performance the objective names. If the objective is to conduct an experiment, have learners conduct one; do not ask questions about conducting one. Untested here; the course-scale version is Constructive Alignment.
  4. Use enough items or tasks for the decision. One confusing item moves a 5-item score by 20 per cent and a 50-item score by 2 per cent (the earlier page's arithmetic). As proposals: a quick check that steers tomorrow's lesson can rest on 3 to 5 items; a grade or placement for an individual should rest on at least 20 selected-response items or three or more performance tasks, across more than one occasion. Where several items hang on one passage or scenario, count the passage, not its items, when judging how much evidence there is: in a simulation of passage-based reading data, treating dependent items as independent overstated reliability, by 4.8–7.4 per cent under a small passage effect and 17.0–21.8 per cent under a larger one (ignoring local dependence overestimates reliability [~M], a simulation of a large-scale English reading test; untested for classroom quizzes). Do not aim for the highest possible internal consistency on a test that deliberately samples separate topics: where items cover distinct pieces of knowledge, a very high alpha may mean the items are measuring something general (reading, diligence, general ability) rather than each topic (reflective modelling can misrepresent domain knowledge [~W]; a high alpha can signal a knowledge test failing its design [~W]; one theoretical article, no data). Judge such a test by its coverage of the table of specifications and by consistency across occasions and markers, not by alpha alone. If the result will be reported per objective (a subscore or a mastered / not-yet flag for each), give each objective items of its own: in a diagnostic physics test, mastery of fine-grained objectives was classified at acceptable accuracy for 19 of 22 objective–test combinations, and the three that fell short were energy objectives that shared about 70 per cent of their items and rested on noisier items (classification accuracy by objective [~W]; why the energy objectives fell short [~W]; one preprint, a research diagnostic, not a classroom quiz); and a state kindergarten readiness assessment's overall score predicted grade 3 achievement about as well as combinations of its subscores (overall score as predictive as subscores [~W], one state report). Where objectives share most of their items, report the total and treat a per-objective reading as a hypothesis to check with a further task.
  5. Strip demands that are not the construct. Have a colleague read each item for wording that could be misread, vocabulary and sentence length beyond the objective, unfamiliar formats and unneeded time pressure. Keep the reading load of a mathematics or science item below the learners' reading level; give a glossary of non-target words, or read items aloud, where reading is not what is being assessed. Basis: on a state fourth-grade mathematics test, some word problems were disproportionately hard for English-language learners compared with non-ELLs of the same overall mathematics score, and a textual analysis traced the difficulty to those items' linguistic features (MCAS word problems showed DIF for ELLs [+M]; linguistic features of the flagged problems [+M]; one large-scale study, abstract only on the claim pages, no effect size), and the same article summarises earlier work finding that linguistic complexity raises item difficulty more for ELLs of equal proficiency (linguistic complexity raises ELL item difficulty [+W], second-hand). It is the nearest the wiki comes to testing this step's premise, on a state test rather than a classroom one; it shows the language demand distorts the score, not that the classroom moves here remove it. A review also argues that verbal tests can understate subject knowledge when their symbolic form does not match how learners encoded it (verbal achievement tests may underestimate subject-matter knowledge [~W], a contention, not a tested result). A second reader catches problems the writer does not see: expert reviewers of draft attitude items found terms used inconsistently and statements with incomplete comparisons (linguist review revealed item communication problems [+W], one developer's account of one inventory). The time-limit and anxiety point is in step 10.
  6. Review content for bias before use. Check every context an item depends on (a sport, a concert, a holiday, a job, the gender of the people in it): either it was taught in class, or every learner can be assumed to know it; otherwise change it. Remove stereotypes (all the doctors men, all the nurses women) and balance the people who appear. Basis: second-hand testimony that females did better on items with female or neutral figures than with male figures, and a count of 42 men to three women in 24 reading passages (male-oriented item content biases test scores [~W], a review of congressional testimony, standardised tests); the offensiveness and unfair-penalisation distinction is from the earlier page (Popham).
  7. Score against written criteria, with calibrated markers. For constructed responses and performances, write a rubric with level descriptors and an anchor example for each level. Before marking, hold a calibration meeting (as a proposal, 30–45 minutes, 3–5 anchor scripts each marked independently, then discussed). Double-mark a sample (as a proposal, 20 per cent or at least 10 scripts in the first cycle) and compare. Basis: three trained raters scoring the same open-ended responses differed significantly in severity (raters differed in severity [~M], one many-facet Rasch analysis); rubric-guided training helped MOOC peer raters place writing samples at the right level more often than no guidance, mainly on subjective, complex criteria and hardly on simple objective ones (rubrics changed peer reviewers' ratings [+M], two quasi-experiments with peer raters, carried to teacher markers).
  8. Look at the items after the first administration, before the scores are used. List each item's percentage correct and flag items most learners missed, items the strongest learners got wrong more often than weaker ones, and items on which one group did much worse than its overall scores predict. Re-score, drop or rewrite flagged items before the grade is final. Treat a group difference as a question to investigate, not a verdict and never as something about the group: a gap by race or ethnicity, gender, socioeconomic status or language background reflects circumstances and treatment, including possible bias in the item itself, which is what this step looks for. Look at the grades you give the same way, not only at test items: in one district's records, grades differed systematically by race, ethnicity and gender among students with the same test scores, attendance, courses and teachers (grade differences by group net of test scores [~M], associational, Chicago high schools). The records cannot say why: a grade carries work habits and course work a test does not, a test carries its own format and timing, and either the marking or the test may treat one group less accurately; the move is to check the marking (blind-mark a sample, compare with the rubric), never to read the gap as a fact about the group. Two bias-detection methods applied to the same state performance assessment agreed on only three of their seven flagged items each (ICC and LDF methods identify different DIF items [~M]), and on one physics test response-time differences between boys and girls did not predict which items were flagged (DIF not predictable from response times [~M]). Both are large-scale testing analyses; the classroom version, with small groups, is an untested proposal and will flag by chance.
  9. Treat decisions near a cut score as uncertain. Where a grade boundary or pass mark falls among many scores, seek a second piece of evidence (another task, a short interview, an earlier result) for learners within a few points of it before deciding. Make that second piece a task or a record, not an impression: 46 secondary teachers' judgements of 406 students' reading comprehension deviated from the students' actual scores by about 22 per cent, mostly overestimating (teachers' judgements overestimated performance [~W], one exploratory study, no effect size). Do not report Cohen's kappa as the consistency of a pass/fail decision. Basis: in a simulation, decision consistency was lowest when the cut score sat at the peak of the score distribution (decision consistency lowest at the distribution peak [~M]); a worked example gave kappa 0.64 at 95 per cent observed agreement (Cohen's kappa paradox [~W], illustrative); an essay argues adjacent scores across a cutoff are not reliably different (grade cutoffs are arbitrary at adjacent scores [~W], argument). Where decision consistency is computed (a common examination, a department pass mark), the three usual methods agreed within 0.044 on normal or near-normal score distributions (DC methods similar for normal distributions [~M]) and differed most with the cut at the mode of a skewed distribution, by 0.101 in a simulation (DC estimates differ most at the mode [~M]), where W-DC or LL-DC are preferred and PS-DC is likely to overestimate (W-DC and LL-DC preferred for skewed data [~M]); all three are one study of simulated and licensure data. Precision also varies along the scale: a review of state tests notes that reliability is usually highest near the mean or the cut and much lower for very high or very low scores (conditional reliability low at the score tails [~M]), so an extreme score is the less precise one.
  10. Set administration conditions to the construct. Use a time limit only where speed is part of what is assessed; keep the order of items the same across versions, or check that a version's later items are not harder because they come later; let anxious learners take low-stakes versions first. Basis: across 223 studies, mathematics anxiety correlated r = −.28 with mathematics achievement (math anxiety degrades performance [~M], correlational, so it does not show that removing time limits helps); an item placed later in one booklet was harder for that booklet's takers, which the authors attribute to fatigue (later-positioned item more difficult [~M], a national examination). On low-stakes checks, and computer-based ones above all, watch for disengaged responding: scoring models assume effortful answers (measurement models assume effortful responding [+W], argument), an answer given faster than the item can be read is a different response process from the one being assessed (rapid guesses are construct-irrelevant [+W]), including such answers is argued to pull scores down (rapid guessing distorts scores [+W], two articles by one author, no data on the page), and uneven rapid guessing often left test events with distorted content coverage (differential rapid guessing distorts content representation [+M], no effect size). Where response times are visible, re-ask or set aside rapid answers before using an individual's score: on low-stakes computer-based tests, a proctor notification triggered by detected rapid guessing was followed by more engagement, higher performance and stronger convergent validity evidence (proctor notification [+M], no effect size printed; rapid guessing signals disengagement [+W]). For a class or school average the risk is smaller: school means on a PISA-based test stayed highly stable despite differential engagement, partly because disengagement pushed individual scores both up and down (school means stable [~M]; distortions cancel [~M]).

Across a term (dose and progression; all quantities are untested proposals). First unit: full table of specifications, a colleague's item review (about 30 minutes per test), a calibration meeting before marking, double-marking of 20 per cent, and item analysis before grades are released. Second and third units: reuse items that were not flagged, rewrite or drop flagged ones, and lower double-marking to 10 per cent if markers agreed on the exact level for roughly four scripts in five in the first round; raise it back to 20 per cent when agreement falls, a new marker joins, or the task changes. (The nearest precedent is a research interview, not classroom marking: a streamlined two-rater procedure reached .96 reliability with 73 per cent agreement, the interviewer's own ratings agreed with final ratings 87 per cent of the time, and rating moved to the interviewer alone with periodic reliability checks; LoU interview reliability [+W].) Each new term or new marker: recalibrate on fresh anchors. Once a year: check criterion validity where the score is used to predict (for example, do autumn scores identify the learners who later needed help?), and revisit the bias review with that year's group results. An item or task moves from "draft" to "bank" after one administration without flags; a score moves from "check" to "grade" only when it rests on the amount of evidence in step 4.

When a condition is not met: if no second marker exists, mark a sample twice yourself a week apart, blind to the first marks, and compare; if there is no time for item analysis, at least look at the three items most learners missed before using the scores; if the group is too small for group comparisons, rely on the content review in step 6.

Fitting the design to a situation

The three facts that most change the decision:

  • What the score will be used for, and how much rides on it. A check that steers tomorrow's lesson tolerates a few items and one marker; a grade, placement or certification needs more items, calibrated scoring and caution at the cut. If unstated, ask: what decision will this result change, for whom, and can it be reversed?
  • Which demands besides the target capability the task makes, and whether they differ across the learners. Language, reading load, cultural reference, speed, anxiety and the response mode are the usual ones. If unstated, ask: who are the learners (languages, literacy, background, access needs), and what must they do to show the capability that is not the capability?
  • Who scores, and how. Selected-response items remove rater variation but narrow what can be assessed; performances and constructed responses widen it but depend on the rater. If unstated, ask: is the response scored by key or by judgement, by how many people, and with what criteria and examples?
If the brief says… Then change… Basis
The goal is a complex-skill or extended performance (a lab, an essay, a presentation, a clinical task) Use a performance task scored by a rubric with level descriptors and anchors; calibrate markers before marking and double-mark a sample; report the score with the number of tasks it rests on Raters differed in severity [~M] (open-ended items, three raters); rubric guidance improved peer raters' placement of writing on complex criteria [+M] (MOOC and graduate peers, carried to teachers); one class's rubric scores varied by a mean of 1.2 of 9 points across six scorers (engineering rubric low scoring reliability [~W])
The goal is verbal-association or a procedure recalled or carried out quickly Use many short selected-response or short-answer items for breadth and reliability; keep the language plain; add a few items that ask for the procedure on a changed case if transfer matters untested; from the earlier page (item-count arithmetic, selected-response trade-off)
Learners are learning in a second language, or have low literacy Keep reading load below the target; gloss non-target words; read items aloud or allow a spoken, drawn or demonstrated answer where reading is not the construct; where it can be done, assess content knowledge in the learner's stronger language as well; do not read a low score as low content knowledge without a second, low-language check; for word problems, have a colleague mark the items whose wording (long or embedded clauses, unfamiliar vocabulary, cultural references) goes beyond the mathematics, and simplify those first State-test word problems with heavier linguistic demands were disproportionately hard for ELLs of the same overall mathematics score (DIF for ELLs [+M]; linguistic features of the flagged items [+M]; one large-scale study, fourth grade); Verbal tests may underestimate subject knowledge [~W] (argument); Spanish–English bilingual kindergarteners did better on mathematics assessments given in Spanish (bilinguals better tested in Spanish [+W], a brief citing one study, no effect size); the fractions example is from the earlier page (Linn & Miller)
Learners are young children (child) Use pictures, objects and spoken instructions; short sessions; observe the performance rather than relying on written answers; more occasions, fewer items per occasion; for dispositions such as curiosity, interest or self-regulation, use an observed behavioural task, not a questionnaire the child completes; expect a single sitting to be unstable, and repeat a measure before acting on one result Achievement moves untested; a short behavioural self-regulation task (Head-Toes-Knees-Shoulders), video-scored with inter-rater reliability of .95–.98, related to teachers' ratings of first graders' social skills and problem behaviour (HTKS validity for first graders [+W]); a preschool number-sense task correlated only .46 with itself three months later, which its authors read as too unstable for young children (moderate test-retest reliability in preschoolers [~W]); both research tasks, carried to classroom observation; a review argues questionnaires are hard to use validly with young children, who have limited reading and cannot yet self-assess abstract states (questionnaires poor fit for young children [+W], about curiosity measures); an argument that on-demand standardised tests alone are problematic for young children (standardised tests problematic for young children [+W], argument)
Learners come from varied cultural or national backgrounds, or a questionnaire has been translated Review every context for taught or shared knowledge; balance people in items; for a translated self-report scale, check its internal consistency and whether its scales relate to each other as expected in your own group before interpreting scores Male-oriented item content biases scores [~W] (second-hand); a Korean translation of a psychosocial measure kept acceptable reliability but its inter-scale correlations were not as expected in 79 immigrant adults (cross-cultural measurement bias in translated instruments [~W]; adults, research instrument, carried to classroom questionnaires); where a Likert questionnaire is repeated to track growth, response styles (such as always agreeing) can shift the estimated slope, differently by construct and scoring model (response styles affect growth estimates [~M]; effects vary by construct and model [~M]; one study of social-emotional surveys)
Learners are anxious about tests, or mathematics anxiety is common in the group Drop time limits unless speed is the target; give low-stakes practice in the same format first; allow scratch work; interpret a score from a timed, high-pressure sitting with caution Math anxiety degrades performance [~M] (meta-analysis of correlations; does not test these moves)
Stakes are high: a grade boundary, pass mark, placement or certification Put more items or tasks behind the decision; seek a second piece of evidence for learners near the cut; report decision consistency rather than kappa; allow a re-take or appeal on a comparable task Decision consistency lowest at the distribution peak [~M] (simulation); Cohen's kappa paradox [~W] (worked example); grade cutoffs arbitrary [~W] (argument)
Time is one session only, or the assessment is a single short quiz Use it to steer the next lesson, not to grade an individual; aim the few items at the one decision you need; combine with other evidence before any grade untested; from the earlier page (a single item's weight in a short test)
One teacher marks for a large group Use selected-response or short-answer items for breadth; reserve judgement-scored tasks for the objectives that need them; mark one question across all scripts at a time rather than one script at a time; re-mark the first ten scripts at the end to check for drift untested proposal
Learners take the assessment in mixed modes: some in the room, some remote, some on phones only Give every learner the same items, aids and time, or record the difference beside the score; avoid long passages, drag-and-drop and wide tables for phone users; compare remote and in-room score distributions before treating them as one set; for a performance, use the same observer checklist live and on video; if remote sittings use automated camera-based proctoring, treat its flags as prompts for a person to review, never as evidence on their own The comparability moves are untested proposals; a systematic review of automated proctoring reports that several studies acknowledged its accuracy varies with learners' demographics, lighting and hardware, which could disadvantage some students (proctoring accuracy varies across demographics [-W], acknowledged limitations, no figures on the page)
Adults in work settings, or the capability is a job task Assess the task as it is done at work, with an observer checklist; use two observers on a sample; check validity by comparing the assessment's ratings with independent observation of practice An interview rating of teachers' use of an innovation correlated .65 with readers' ratings of written ethnographic protocols, which its authors read as support for its validity (LoU interview validated against observation [+W]; teachers, research instrument; the way validity was checked is carried, not a result about learners)
Peers mark each other's work and the marks count toward a score Count peer marks only where the marking scheme leaves little to judgement (exact answers, stated point allocations) and each script has several peer markers; have the teacher mark a sample and compare before using the marks; for subjective or complex criteria, train peers on a rubric with anchors first, or use peer marks as feedback rather than a grade; after a calibration round, show each peer marker how their marks compared with the teacher's (too generous, too harsh) before they mark for real; expect markers to be least accurate on work far above or below their own, so mix the levels of each script's markers In one online course, peer marks on two exact-science exams did not differ significantly from the teacher's and correlated .88–.91 with them (peer assessment equivalent for exact-science exams [~M]; one course, a clear marking scheme, r1; non-significance is not equivalence); rubric training helped peer raters most on complex criteria (rubric guidance for peer raters [+M]); in a MOOC experiment, telling students about their own grading bias was followed by more accurate peer grading (feedback on grading bias increased accuracy [+W], no statistic printed); in MOOC peer-grading data, higher-scoring students were generally more reliable graders, and students were least biased grading work near their own level, the strongest marking the weakest work down and the weakest marking the strongest up (grader level and grading reliability [~W]); a review reports adequately reliable peer assessment of 187 medical interns' professional performance (peer assessment of professional skills [~W], second-hand)
A self-report questionnaire (strategies, study habits, self-efficacy, attitudes) is used alongside achievement measures Read its scores as learners' perceptions, not as what they did; ask about a specific task straight after it rather than about general habits; check a sample against a trace of what learners actually did (their work, a think-aloud, logs) before acting on it; check reliability in your own group, subscale by subscale rather than for the total, and treat short (two- or three-item) subscales as the likeliest to fall short; do not read a test of what learners know about strategies as a record of what they do; do not use it for an individual high-stakes decision Across published online self-regulation questionnaires, alphas of relevant scales ranged from .52 to .95 and all but two instruments had at least one scale outside .70–.90 (existing online SRL scales show reliability problems [~W], the authors' review); in the new questionnaire's own validation, nine of ten subscales exceeded .70 and the three-item one fell to omega .665 (SRL-O internal consistency [~W]); a scenario-based test of strategy knowledge correlated moderately with a strategy questionnaire but low and mostly non-significantly with a microanalytic measure taken during a learning task (convergent with the questionnaire [~W]; weakly related to the online measure [~W]; one instrument, college students). Arguments, no data on the pages: off-line self-reports rest on reconstructing earlier processes from memory, with more distortion the longer the delay (off-line self-reports and memory reconstruction [~W]); the questions themselves can prompt reports of strategies never used, unequally across learners (prompting distorts retrospective self-reports [~W]); questionnaires may capture perception rather than performed strategies (questionnaires measure perception [~W]); a review reports internal consistency of .62–.94 for self-efficacy measures, which tracked domain measures and not general ones (self-efficacy measures reliable and domain-specific [+W]), so a well-built, domain-specific scale can be consistent
The assessment is embedded in a game or simulation (stealth or in-game assessment) Treat each kind of in-game item as an untested instrument: before using the scores, compare each item type with an established measure of the same thing in a pilot group, keep the item types that agree and drop or rework the rest; check every target dimension separately, since one can fail while others work; check the reference measure's own reliability in your age group before treating disagreement as the game's fault; where the game's model predicts learner states, compare its accuracy across groups before using it In one problem-solving game, the stealth assessment's facets reached alpha .76 with middle-school students while the established reference test (MicroDYN) reached only .67 and .43, which the authors put down to the tasks being hard for that age (stealth facets reliable, reference test not [~W]); a review reports deep-learning stealth assessment predicting middle-schoolers' science knowledge from gameplay, with a demographic bias audit and debiasing without loss of accuracy, and AUCs of .85–.91 across studies (stealth assessment accuracy and bias audits [~W], second-hand). In one profiling game, questionnaire-like in-game items (diary ordering, ranking, narrative questionnaires) matched a reference interest inventory best, diary ordering 78 per cent of the time, while other item types managed 10–22 per cent (item types differed in correlation with the questionnaire [~W]), and most items for one interest type did not work (enterprising items failed [~W]); vocational interests, not achievement, carried here as a validation procedure
The work is a product or performance for a real audience Score it against the criteria by markers who have calibrated, separately from the audience's reaction; record audience response as its own observation, not as the grade untested proposal; the audience relationship is owned by Authentic Audiences and Purposes
The test uses several items on one passage, case or data set Count passages, not items, when judging whether there is enough evidence; estimate consistency at the passage level Ignoring local dependence overestimates reliability [~M] (simulation of a large-scale reading test)
Several versions of the test exist (shuffled order, A/B forms) Keep item order fixed across versions, or compare an item's difficulty across positions before treating versions as equal Later-positioned item more difficult [~M] (national examination; one item)
The score will be used to place learners or to predict who will need help later Do not place on one test alone: combine it with course grades or a second measure; use the full scale score rather than a pass or proficiency label; check locally which components actually predict the later outcome and drop or down-weight those that do not; where a cut on one measure (a screener, an English-proficiency score) is meant to signal readiness for another, set it from local records of the share of learners at each score who went on to succeed, not by convention; review placement and identification rates by group, and check that the measure predicts as well for each group; a measure on which most learners sit near the top cannot separate them, so do not place on it Among Chicago graduates, a standard deviation of high school GPA was associated with a 15 percentage point rise in four-year college graduation against 7 for test scores (GPA predicts graduation more than tests [+M]); in another Chicago analysis GPA's link to graduation held across high schools while ACT added nothing once GPA was in the model (HSGPA strongly predicts graduation [+M]; HSGPA slope consistent across schools [+M]; ACT a weak predictor [~M]; ACT adds little beyond HSGPA [~M]); ninth-grade GPA was the most predictive single indicator for every English Learner group (GPA most predictive for English Learners [+M]); diagnostic topic scores predicted algebra proficiency beyond a prior state test (MDTP topic scores predict proficiency [+M]); a placement test's multiple-choice part predicted passing as GPA did while its writing part predicted nothing (placement components differ in prediction [~M]); a practice guide cites 24–33 per cent of college entrants misplaced by single tests (misplacement rates [~W], second-hand); a narrative review attributes under-identification of diverse students for gifted programmes to the measures (biased identification measures [~W], argued, not tested). Second-hand reports of the same Chicago work: freshman-year GPA alone identified 80 per cent of on-time graduates against 65 per cent for eighth-grade tests and risk measures together (freshman GPA predicts on-time graduation [+W]), and among students with the same ACT score college graduation was 77 per cent at a GPA of 3.5 or more against 40 per cent at 2.0–2.4 (GPA outpredicts ACT at equal scores [+W]); studies finding grades less predictive are said to rest on restricted samples or self-reported or weighted GPAs (restricted samples explain divergent findings [~W], an interview, no data). By group: in Arkansas records almost all middle and high school indicators predicted later outcomes about equally well across student groups, with exceptions for science and English language arts proficiency in middle school and science proficiency in high school (predictive strength similar across groups, with exceptions [~W]); a practice guide reports that SAT mathematics scores underpredict women's performance in college mathematics and that girls scored lower than boys on timed NAEP tests at the same course level (timed-test gap and underprediction [~W], second-hand, no effect size), so a test can mispredict for a group whose later performance it is used to forecast; in advanced university STEM courses, school-leaving grades clustered near the maximum (means about 93 of 100) did not account for later differences between groups, which the authors read as a ceiling effect (school grades show a ceiling [~W], an interpretation). A logistic-regression approach estimates, for each English-proficiency score, the probability of reaching proficiency on the content test, as a way to set an English-proficient cut (half-probability criterion [~W], a method described in a federal evaluation, no result on the page). All associational, mostly US secondary and college records, carried to classroom placement
Results will be compared across groups (gender, race or ethnicity, language background, family income), or a gap between groups has been noticed Treat the gap as a question about the measures, the marking and the teaching, never as a property of the group; compare the group's test results with its grades, since the two can disagree; blind-mark a sample of work with names removed and compare with the marks given; check the items for language load, untaught background and time pressure (steps 5, 6, 8, 10); where a score is used to predict or place, check that it predicts as well for each group; with small groups, wait for a second administration before concluding anything Grades differed by race, ethnicity and gender among Chicago students with the same test scores, attendance, courses and teachers (grade differences by group net of test scores [~M]), while a practice guide reports girls scoring lower than boys on timed NAEP tests at the same course level and SAT mathematics underpredicting women's college mathematics (timed-test gap and underprediction [~W], second-hand): grades and tests disagreed, in different directions for different groups, which is why neither is read alone; language load produced item-level gaps for English-language learners (DIF for ELLs [+M]); most indicators predicted equally well across groups in one state, but not all (predictive strength similar across groups, with exceptions [~W]). All associational records; none tests the classroom moves
AI or an LLM will score constructed responses Use it as a second reader under a teacher's control, not as the grader of record; before relying on it, compare its marks with teachers' on a sample of your own scripts, dimension by dimension, and check for systematic leniency or severity; send mid-range responses and those from learners with weaker written language to a teacher; where repeat runs differ, score more than once and take the median; write the rubric so it names what counts as evidence; judge the scorer's disagreement with teachers against how far two teachers disagree on the same scripts, not against perfect agreement; check that it uses the whole scale rather than marking low or bunching marks in the middle; if it runs reasoning and fast modes, use the reasoning mode; if an LLM conducts a conversational assessment, read sample transcripts for turns that give the answer away before scoring them Agreement fell sharply on mid-range short answers while the extremes were scored well (mid-range degradation [~M]); lower linguistic quality raised the odds that every one of eleven AI approaches underestimated Grade 9 physics understanding (AI scoring underestimates linguistically weak explanations [-M]); repeated ChatGPT sessions on the same essays reached Krippendorff's alpha 0.625 (limited reproducibility [~M]) while its best single-pass agreement with examiners was weighted kappa 0.718 (strongest examiner agreement [~M]); the best of many models reached ICC 0.470 with programming teachers, and models split into lenient, strict and middling graders (moderate agreement at best [~M]; grading styles differ [~M]); repeat-run reliability ranged widely by model (reliability stratified by model [~M]); prompting strategy changed agreement and direction of bias, and agreement differed by rubric dimension (Fs+CoT strongest agreement [~M]; RAG systematic leniency [-M]; strategy-specific bias [~M]; dimension-specific agreement [~M]); a revised workflow raised exact agreement on physics question parts from about 63 to about 70 per cent, with partial credit hardest (question-part agreement [~M]); revising a rubric to state its constructs raised human–machine agreement (rubric revision improves alignment [+M]); a review argues AI analyses may perform unevenly across student groups (uneven performance across populations [~W]). On one computer-science exam double-marked by human pairs, whose totals differed by a mean of 2.61 of 35 points, six closed-model configurations came within that human-to-human difference, the best at 1.64 (LLM graders within the human floor [~M], preprint), which is the comparison this row asks for; five LLMs scoring IELTS essays mostly marked lower than human raters, some also compressing scores into a narrow range (systematic underestimation [~W]); four LLMs grading 32 public-health exam essays agreed poorly with human grades in fast mode (weighted kappa 0.07–0.29) and better in thinking mode (fast mode limited, thinking mode better [~W]); models grading the same open-ended answers differed significantly in leniency (grades differ across models [~W]); a single model judge re-run three times gave nearly identical scores, so consistency with itself says nothing about agreement with teachers (within-judge stability [~W], scoring generated videos, not learner work); an older automated essay scorer agreed with human raters 94.6–98.2 per cent but needed many pre-scored essays to calibrate, which a review judged impractical for most small classes (AES agreement and calibration constraint [~W], second-hand); LLM-run assessment conversations were observed to give away more than intended and to be harder to score than scripted ones (LLM conversations leak information [~W], one case study). Mostly single studies of 2025–2026 models, many preprints
Learners may use generative AI on work done outside supervision Name the construct the task targets; for recall or routine procedures, assess under supervision; pair a take-home product with a supervised twin (a short oral explanation or an in-class task on the same capability) and read the two together; do not treat AI-detector output as evidence for an integrity decision, least of all for second-language writers; separate permitted language support from substitution in the rules An evidence-centred argument that AI contribution breaks the link from a product to the learner's proficiency, almost completely for recall and routine tasks (GenAI disrupts evidence rules [~W], argument); an argument for assessment twins (assessment twins [+W], argument); an LLM detector was poor on short answers though better on programs and longer reflections (self-detection task-dependent [-M]); a review cites a discontinued classifier's false positives and detector bias against non-native writers (AI detection unreliable and biased [-W], second-hand); another review reports a benchmark in which seven detectors flagged authentic TOEFL essays by non-native writers as AI-generated at a mean false-positive rate of 61.3 per cent, far more than native writers' essays (61.3 per cent false positives for non-native writers [-W], second-hand); in a large sample of Reddit education discussions, students described detectors as linguistically biased, unreliable and distressing (detectors reported unreliable and biased [-W], reports, not a test of detectors); undifferentiated rules are argued to turn language disadvantage into integrity risk (undifferentiated GenAI rules [-W], argument)

Seventeen of the twenty-one rows cite a claim; none of those claims tests the classroom move in its row (the nearest are that revising a rubric raised human–machine agreement, with an AI scorer, that peer marks matched a teacher's on exams with a clear marking scheme, and that feedback on their own bias made MOOC peer graders more accurate). They come from large-scale examinations, simulations, research instruments, administrative records, AI-scoring studies, peer raters in online courses or a single class, and are carried to classroom assessment. The single-session, large-group, real-audience and recall rows are untested proposals; the mixed-mode row's comparability moves are too, and only its proctoring caution has a basis (a review's acknowledged limitation); the young-children row rests on arguments and two research tasks, not on classroom achievement assessment.

Observation, state and explanation

What can be observed is a response to a stated task under stated conditions (time, aids, mode, language, who scored it, with what criteria). "Understands fractions", "is weak at writing" or "the test is biased" are inferences. A score is evidence about the capability only after the alternatives in the table have been considered.

Response Plausible explanations Observation that could change the interpretation
A learner scores low on a written test but explains the idea well in class Reading or language load, not the capability; anxiety in the test setting; the test sampled content the learner had not been taught; the class explanation was prompted Give a matched item orally, or with a glossary and no time limit; check the item against the table of specifications.
Two markers give the same script levels that differ by more than one Criteria too vague at that level; one marker is consistently harsher; the script mixes strong and weak features the rubric does not weigh Have both mark three anchor scripts blind; see whether the gap is constant across scripts (severity) or varies (criterion ambiguity).
One group does much worse on a few items than its overall scores predict The items draw on background not taught in class; wording or format unfamiliar to that group; chance, in a small group Read the items for outside knowledge and stereotypes; re-administer a version with a changed context; check whether the pattern recurs next administration.
Strong learners get an item wrong more often than weaker ones The key is wrong; the item is ambiguous and strong learners see a second reading; the item rewards a guessing strategy Check the key; ask two strong learners how they read it.
Scores are high on recall items and low on the performance task for the same objective The recall items did not sample the capability; the performance task adds demands beyond it (time, materials, writing); the learner has knowledge without the skill Compare with a short supervised performance with reduced extra demands.
Many scores fall just either side of the pass mark The cut sits where scores bunch, so many decisions would change on a re-test Add a second piece of evidence for learners near the cut before deciding.
A learner's score on a low-stakes computer-based check is far below their class work, and many answers took a few seconds Disengaged responding (rapid guessing), not the capability; the content was not taught; anxiety or a device problem Look at response times item by item; re-ask the rapidly answered items with the teacher present; compare with a short task done in class.
English-language learners do much worse than other learners on the word problems of a mathematics test but not on its computation items The wording of those problems, not the mathematics; the problem contexts draw on unshared background; a real gap in the problem-solving skill Give the same problems rewritten in plain language, or read aloud or in the learner's stronger language, and see whether the gap closes; compare with a problem-solving task done orally in class.
A learner's questionnaire reports frequent use of a strategy their work shows no sign of The report reflects what the learner believes or thinks is wanted, or a reconstruction from memory; the strategy is used in other tasks; the work does not show the strategy even when used Ask about this task straight after it; look at the work or a think-aloud on a similar task.
One group's grades sit lower (or higher) than its test scores would predict, or the reverse The grade carries work habits, completed work or behaviour the test does not; the marking treats that group's work differently (possible bias in judgement); the test is less accurate for that group (timing, language load, background, a ceiling); chance, in a small group Blind-mark a sample with names removed and compare with the marks given; compare timed and untimed or low-language versions of a few items; look at which grade components (homework, tests, participation) carry the difference; check again next term before acting.
An AI-scored response gets a much lower mark than a teacher would give The response is mid-range or its language is weak, where AI scoring is least accurate; the rubric rewards surface wording; the response really is weak Have a teacher mark it blind; check whether the gap recurs for the same learner or the same kind of response.

These rows are proposals, untested with teachers or learners as diagnostics. Each probe is a further assessment with its own error, so record it and keep explanations open rather than choosing the one whose name fits.

Evidence and qualifications

No claim in the wiki tests whether classroom assessments built with these moves lead to more accurate decisions about learners. The claims below test parts of the relationship, mostly outside the classroom.

The learners, instruments and settings differ (large-scale examinations, simulations, peer raters online, research questionnaires), so do not rank these by effect label. None measures the classroom decisions this page is about.

The evidence by question

Further evidence, not yet read against this model

Claims linked to this page that do not bear on its relationship, a step or a situation row (reread 2026-10-15 and 2026-10-19, and a second pass on 2026-10-19 over the claims the orphan-claim linker added that day; the rest were worked into the model above). Each says what it is and how far it bears on the page.

Objective and learner-valued goal

The designer's objective here is a defensible decision: a score that supports the conclusion drawn from it, for each learner it is drawn about. Learners value something related but different. Some want a fair grade, some want to be seen as able, some want the result for what it opens (a course, a job, a licence), and some want to know what to work on. Ask them, before a high-stakes assessment, what they need the result for and what they expect it to show; ask afterwards whether it showed what they can do, and if not, what got in the way. A learner who says "I knew it but couldn't read the question" or "I ran out of time" is reporting a possible construct-irrelevant demand; record it as an observation to check, not as an excuse or a finding.

For example, an adult nursing assistant may want a certificate that lets them take on medication rounds, while the designer's objective is a reliable score on a written dosage test. A high written score supports the designer's inference about calculation; the learner's goal needs an observed performance on the ward as well. Record both, so that a test score is not read as the capability the learner came for.

What would revise this model?

The expectation should weaken if classroom studies find that decisions made from assessments built with a table of specifications, more items, calibrated scoring and bias review are no more accurate (against an independent judgement of the learners' capability) than decisions made from ordinary teacher-made tests, or that the gain does not change what happens to learners next. If calibration meetings do not reduce differences in teacher severity, step 7 should drop to double-marking alone. If content bias review in classrooms does not change group differences, the review stays as a fairness duty, not an accuracy claim. If low-language and untimed versions do not change the scores of second-language or anxious learners, the construct-irrelevant explanation should lose weight for those groups. Do not protect the model by calling every unexpected score "invalid" after the fact; the checks have to be made before the scores are used.

Content validity, construct validity, criterion validity, consistency across occasions, items and raters, and group fairness are separate claims, each needing its own evidence. The present evidence establishes no minimum number of items, no acceptable level of rater agreement for classroom marking, and no amount of calibration that is enough.

Claims

  • Criterion- and Norm-Referenced Testing — validity and reliability apply to standardized as well as teacher-made assessments, with additional considerations
  • Research Data Collection Methods — the same reliability/validity distinction, applied to research instruments and constructs rather than classroom assessments specifically
  • Standardized Test Fairness and Bias — documented, large-scale cases of the bias and validity failures this principle describes in general form
  • Assessment for Learning — validity and reliability are prerequisites for assessment evidence to be trustworthy enough to act on
  • Competency-Based Assessment — applies these properties to a criterion-referenced judgement against a threshold, and models assessor agreement and threshold setting for that case

Examples

Key Sources

  • Linn, R. L., & Miller, M. D. (2005). Measurement and assessment in teaching (9th ed.). Pearson.
  • Popham, W. J. (2005). Classroom assessment: What teachers need to know (4th ed.). Allyn & Bacon.
  • Arduini-Van Hoose, N. (2020). Selecting high-quality assessments; Methods of data collection. In Educational psychology. Retrieved from https://edpsych.pressbooks.sunycreate.cloud. CC BY-NC-SA 4.0.
  • Thomas, Mollie B. (1975). Underlying Constructs of Locus of Control of Reinforcement. https://eric.ed.gov/?id=ED108039
  • Wolkowitz, A. A., & Smith, R. (2024). A Practical Comparison of Decision Consistency Estimates. Practical Assessment, Research & Evaluation, 29(6). https://pareonline.net/getvn.asp?v=29&n=6
  • Dozier, C. L. (2008). Literacy coaching: Engaging and learning with teachers. The Language and Literacy Spectrum, 18. https://eric.ed.gov/?id=EJ1059644
  • Haller, Archibald O. and Miller, Irwin W. (1963). The Occupational Aspiration Scale: Theory, Structure and Correlates. Michigan State University Agricultural Experiment Station. https://eric.ed.gov/?id=ED016712
  • Frogner, Ellen A. (1969). Using the Language Inquiry as a Teaching Device. Illinois State-Wide Curriculum Study Center in the Preparation of Secondary English Teachers (ISCPET). https://eric.ed.gov/?id=ED034768
  • Martiniello, M. (2008). Language and the Performance of English-Language Learners in Math Word Problems. Harvard Educational Review. https://doi.org/10.17763/haer.78.2.70783570r1111t32
Page metadata
Field Value
type principle
id validity-reliability-and-bias-in-classroom-assessment
aliases balance-situations-sources-control-items, dc-estimate-guideline-085, side-by-side-assessment-analysis-with-teachers, use-oas-for-loa-research-and-counseling
title Validity, Reliability, and Bias in Classroom Assessment
description A classroom assessment score is expected to support a decision about a learner only as far as the tasks sample the intended capability without demands unrelated to it, the score is consistent across items, occasions and raters, and no group is disadvantaged by content or administration; no claim here tests that relationship in classrooms, and the claims that bear on it come mostly from large-scale testing and simulations.
status review
moved_from pattern/expert-panel-consensus-item-selection-pattern
canonical True
generated by claude/unspecified on 2026-10-19
trust unverified — no one has confirmed this page's content
sources 5 — thomas-1975, wolkowitz-2024, dozier-2008, haller-1963, frogner-1969

Frontmatter is the page's machine-readable record — see the Schema & Guide.