Constructed-Response Assessment Items

Element · All elements
Evidence · 51 claims (16 for, 33 mixed, 2 against) · 39 studies (14 design, 11 associational, 7 causal, 3 review, 2 theoretical, 1 quant-synthesis, 1 qualitative), q1–q3 · 7 of 39 report an effect size · 51 claims rest on one study

Description

Constructed-response items ask students to recall information and produce their own answer rather than recognize a provided one, which reduces guessing compared with selected-response items. Two major kinds are used: completion/short-answer items (answerable in a word, phrase, number, or symbol — the same underlying format whether phrased as a statement to complete or a question to answer) and extended-response items (answers ranging from a paragraph to several pages, often called essay questions when they run long).

Completion and short-answer items are easy to construct and are sometimes called "objective" on the assumption that only one answer is correct — but unless a question is phrased very precisely, multiple genuinely correct answers often exist ("Where was President Lincoln born?" can correctly be answered "in a log cabin," "on Sinking Spring Farm," "in Hardin County," or "in Kentucky"), which undercuts that objectivity in practice. Common errors: more than one defensibly correct answer; too many blanks in one item, making it confusing rather than difficult in the intended way; and blank-length or spacing that inadvertently cues the expected answer.

Extended-response items are the most adaptable format for measuring complex learning outcomes — particularly integration and application — and, because they require writing, double as a way to assess writing skill directly. They have a reputation for being easy to write, but a genuinely well-constructed item (one clearly worded, tied to the actual learning outcome, and capable of assessing complex thinking) is hard to devise. Common errors: an ambiguous question that gives students no real guidance on what's being asked (leaving them free to answer "yes" or "no," or to address only one of several plausible dimensions of the topic); scoring without a rubric, so the same teacher scores the same response differently depending on unrelated factors (unexpected responses, irrelevant information, grammatical errors); a "halo" effect from grading one student's whole test at once rather than one question across all students at once, so accumulated impressions of a given student bleed into how later answers on the same paper are scored; and offering students a choice of prompts, which undermines score comparability since students are no longer really taking the same test.

The central disadvantage of extended-response items is unreliable scoring: different teachers can score an identical response differently, and the same teacher can score the same response differently on different occasions (Linn & Miller, 2005). Several concrete steps improve scoring reliability and validity: (1) write an outline of a model answer before scoring begins, to make explicit what students are actually expected to include; (2) read a sample of student answers first, to calibrate expectations and notice common misconceptions before scoring in earnest; (3) decide in advance how irrelevant information and mechanical errors (grammar, spelling) will be handled, rather than deciding ad hoc per paper; and (4) score using either point-scoring or a scoring rubric, rather than an unstructured overall impression. In point-scoring, specific components of an answer are each assigned a point value — useful for consistency, but prone to pulling attention toward easily-counted facts (e.g., naming risk factors) at the expense of the higher-level thinking the item may actually have been designed to assess. A scoring rubric that describes the quality of an answer at each performance level is generally the stronger alternative when higher-order thinking is the real target — see Rubrics.

Design Implications

Context

Requirements

• A genuinely clear prompt: well-constructed extended-response items make the actual task explicit through wording and, often, planning notes or hints, not through length or elaborateness alone
• A model answer and a scoring plan (rubric or point-scoring) prepared before scoring begins, not improvised paper by paper

Constraints

• Completion and short-answer items outside mathematics are poorly suited to complex learning outcomes and are frequently harder to score reliably than they first appear, since genuinely unambiguous phrasing is difficult to achieve
• Extended-response scoring reliability degrades specifically when there is no rubric, when papers are graded question-by-question-per-student rather than one question across all students, and when choice among prompts is offered
• Point-scoring's ease of use comes at a real cost to validity when the actual goal is higher-order thinking rather than fact enumeration

Target Learners

• All students in a K-12 or classroom-assessment context; extended-response items are the natural choice when the learning goal itself requires integration, application, or written expression that selected-response items structurally cannot assess

Target Learning Goals

• Recall and production of specific factual content (completion/short-answer); integration, application, and writing skill (extended-response)

Affordances

• Rubrics — the recommended scoring structure for extended-response items when higher-order thinking, not just factual coverage, is the target
• Validity, Reliability, and Bias in Classroom Assessment — the scoring-reliability problem this page describes is a direct instance of that principle's reliability concerns

Claims

• Adding writing prompts and essays to state English language arts assessments shifted classroom instruction toward writing [~W]
• Despite generally accurate feedback, the LLM's scoring explanations could be unpredictable and prone to logical inconsistency, failing to award points even when citing the correct rubric directive [-W]
• LLMs overrate short underdeveloped essays and over-penalize minor language errors in well-developed essays [~W]
• Adding a model answer to the GPT-4 prompt was counterproductive, lowering scoring reliability [~W]
• Different LLMs rely on different subsets of rubric traits when assigning essay scores [~W]
• For short-answer questions, real student responses received higher LLM-likelihood estimates than LLM-generated responses, a reversal of the pattern in other tasks [~W]
• Students perceive AI scoring as accurate for grammar, mechanics, and organization but skeptical of its ability to evaluate diverse styles, complex writing, tone, and emotion [~W]
• Admissions staff rate essays they perceive as AI-generated lower than essays they believe are human-written, even after adjusting for measured essay quality and actual AI authorship [-W]
• The revised Round II grading workflow reduced AI over-awarding and total-score MAD, most clearly for OE1 (7.1% to 4.8%) and QM (9.4% to 3.8%) [~W]
• Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades [~W]
• Newer LLM generations such as GPT-4 generally outperform GPT-3.5 in short-answer grading accuracy, but this advantage is not consistent across all tasks and domains [+W]
• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades [~W]
• AI-assigned total scores on a handwritten chemistry exam align strongly with TA-assigned totals (R2 = 0.91), while per-problem agreement is lower (R2 = 0.61–0.85) [+W]
• The GED essay is scored holistically on a six-point scale by two independent readers [+W]
• Human inter-rater reliability with a single rubric exceeded the ±5-point equivalence margin used for rubric-source comparisons [+W]
• A page-by-page, rubric-guided multimodal LLM workflow graded an authentic handwritten exam in about three hours at roughly $100 in token costs, versus about $3,500 in TA time [~W]
• Adapting the scoring mechanism substantially reduced ceiling effects found in the pilot study [~W]
• Multiple-choice single-select items are less difficult than open-ended response questions, and delayed perceived task difficulty ratings relate more strongly to relative item difficulty for open-ended items [+W]
• Automated essay scoring can achieve high agreement with human raters but faces calibration constraints [~W]
• Transformer-based NLP models achieve automated essay scoring with correlations to human grading exceeding 0.85 [+W]
• AEEGS and human raters produced nearly equivalent, low mean scores (5.94 vs 5.97 of 20), with AEEGS slightly higher [~W]
• Six closed-model configurations grade at or below the human inter-grader floor of 2.61/35 on the CV exam, with the best reaching MAE 1.64 [+W]
• A construction-focused correctness rubric underperformed behavior-focused and undefined prompts for LLM scoring of code explanations [+W]
• Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading [+W]
• Equivalence failures under human evaluators were confined to five assignment-by-question-type cells, four of them on code questions [~W]
• Grading time shows a sweet spot: students grading around 20 minutes (time z-score near -0.25) have significantly lower residual standard deviations than slow graders, while snap graders are unreliable and inflate grades [~W]
• Human scoring and feedback for HDR takes about 11 minutes 30 seconds per learner, while GPT-based scoring and feedback is nearly instant [+W]
• Hint-based prompts raise ChatGPT scoring agreement with human experts to 98–100%, while no-hint prompts are unstable (Q2 precision 0.42) [+W]
• Human evaluators' prioritization of essential features was significantly better for low-quality essays than for average- and high-scoring essays [~W]
• In one study of five LLMs scoring 60 undergraduate essays on IELTS rubric dimensions, systematic underestimation of human scores was the dominant bias trend, with some models also compressing scores into a narrow range [~W]
• LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes [~W]
• A short strict-grader preamble knocks 14 of 17 open-weights LLM graders out of the graded band on a computer-vision exam, with three stopping grading outright [~M]
• Machine–machine agreement exceeds human–machine agreement but remains below human–human agreement [~W]
• A multi-armed bandit controller achieves scoring accuracy comparable to exhaustive grid search while reducing LLM calls by 78.4% and token consumption by 72.8% [~W]
• NLP-based feedback generation for textual student answers in large courses reduced grading effort by up to 85% with high precision (reported from Bernius et al.) [~W]
• Native speaking fourth graders answered more open-ended text comprehension questions correctly than non-native speakers (Study 2) [~W]
• Benchmarking shows NLP-based automated essay scoring achieves grading times up to 80% faster than human graders while maintaining comparable accuracy [~W]
• Ensembles of operational raters outperform the LLMs: two raters suffice on average across aspects, but larger ensembles are needed for individual aspects [~W]
• Rubric-enhanced prompting (Variant 2) raises LLM assigned scores relative to the no-rubric baseline, with model-specific gaps at particular taxonomy levels [~W]
• On assessment item Q10, describing a TinkerPlots plot, 32% of students achieved the maximum score of 4 and 70% achieved half marks or better [~W]
• Removing detailed rubric explanations from prompts improved grading accuracy across all approaches in an ablation study [~W]
• Word count correlates with suggested final grade, with the ordering Gemini Pro > humans > ChatGPT > Gemini among statistically significant correlations (human Spearman's rho = 0.603) [~W]
• Open response format is associated with higher Alpha than multiple choice in knowledge tests [+W]
• Cognitive artifact scoring in the digital workbook achieved high intercoder reliability, with average Spearman's rho of 0.97 overall [+W]
• Grading variability: 40.7% of grade variance lies between teachers and essay quality is the largest random effect [+W]
• Experiential attitudes toward feedback predict agreement with expert criteria on weak-essay grades [~W]
• After studying the composition scale, normal-school students double-rated 32 seventh-grade compositions, each composition being rated by two students with the second rater rating a different set [+W]
• Sentence units of a completion test differ greatly in completion time, ranging from 12.8% of total test time for unit 1 down to 3.4% for unit 11. [~W]

The evidence by question

• Which practice format: short answer, multiple choice, or matching the final test? Evidence on whether initial short-answer questions produce more learning than initial multiple-choice questions is mixed, with recent studies finding little or no difference [~M]
• Do markers scoring the same responses agree, and do written criteria bring their judgements closer? Raters differed significantly in severity/leniency in the MFRM analysis [+M]
• Can a language model score learners' written constructed responses as a trained human marker would, and on which tasks? LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks [~M]

Related Elements

• Selected-Response Assessment Items — the complementary item family for recognition-level goals
• Computer-delivered figural response test items for architecture

Examples

• A third-grade mathematics extended-response item asking students to divide books among classrooms and explain their reasoning in words (Illinois Standards Achievement Test, 2006)
• A fifth-grade science item asking students to plan a full investigation (prediction, materials, controlled/manipulated variables, procedure) rather than just answer a single question (Washington State Assessment of Student Learning, 2004)
• Give learners control over how learning is measured via assessment choices and science-fair-type projects

Key Sources

• Linn, R. L., & Miller, M. D. (2005). Measurement and assessment in teaching (9th ed.). Pearson.
• Arduini-Van Hoose, N. (2020). Teacher-made assessments. In Educational psychology. Retrieved from https://edpsych.pressbooks.sunycreate.cloud. CC BY-NC-SA 4.0.

Page metadata

Field	Value
type	element
id	constructed-response-assessment-items
title	Constructed-Response Assessment Items
description	Completion, short-answer, and extended-response (essay) items ask students to recall and produce an answer rather than recognize one, reducing guessing but introducing a genuine scoring-reliability problem that model answers, sample-reading, and rubrics are all meant to address.
status	review
canonical	True
generated	by claude/unspecified on 2026-08-29
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.