Skip to content

Collaborative Evaluation

Pattern · All patterns
Evidence · 21 claims (7 for, 14 mixed) · 41 studies (15 causal, 7 quant-synthesis, 7 associational, 5 theoretical, 3 review, 3 qualitative, 1 design), q1–q4 · 5 of 41 report an effect size · 7 claims rest on one study

Description and scope

A reusable policy for learning to judge quality with others: an object to evaluate (sources, drafts, proposals, solutions, performances) and a small set of criteria → each learner judges it alone, in writing, with a reason tied to a criterion → the judgments are revealed together and compared in a small group → disagreements are settled by pointing to evidence in the object, or recorded as reasoned disagreement → the group's judgment is compared with a reference judgment (the teacher's, an expert's, or the known quality of an anchor exemplar) → each learner judges a new object alone, or revises their own work against the same criteria. The intended change is in each learner's evaluative judgment: how closely their own judgments of new work agree with a reference (assessment-accuracy), whether they can justify a judgment from the criteria (conceptual-understanding), and, where they then revise their own work, its quality (skill-performance, general-achievement).

It sits inside two converted pages and does not repeat them. The collaborative learning pattern owns joint work in general: what each member can do afterwards is the target, a group product is not, and members are checked individually. This page applies that rule to one kind of task, evaluation against criteria, where a group verdict is especially easy to mistake for each member's judgment. The Assessment for Learning principle owns the learner-side reading of evidence against shared criteria, including how well calibrated self-assessment is; this page is one configuration for building that capability, by judging other work before one's own. Structured academic controversy is the neighbouring configuration for a contested question, where the aim is a reasoned position, not a judgment of quality. Peer instruction has the same "answer alone, discuss, answer again" shape for concept questions.

No claim in this wiki tests collaborative evaluation as such: none compares learners who evaluate together and reach a group judgment with learners who evaluate alone, or with a teacher's judgment simply given. The claims that bear on it are of four kinds: peer assessment against no assessment (learners judging each other's work, alone or with discussion); rubric guidance for peer raters; the accuracy of learners' own and peers' judgments; and peer discussion of concept questions followed by teacher explanation. Each is carried to this pattern by extrapolation, and the page says how far. The sequence below is the earlier page's five steps with a reference comparison and an individual check added; each step carries its evidence status, and the response-dependent policy is an untested design proposal.

Inputs: establish the design brief

Field Record or elicit
Learner and starting observation Before any discussion, each learner's written judgment of one or two objects, with the criterion each reason rests on and their confidence. Compare it with a reference judgment. Record prior knowledge of the domain the objects come from, age band, language of instruction and any status or friendship differences in the groups. Preserve unknowns rather than labelling learners novice.
The evaluation object What is judged (a source, a draft, a proposal, a solution, a performance), whether a defensible reference judgment exists for it, whether quality turns on correctness (a calculation, a factual claim) or on qualities informed people weigh differently (an argument, a design), and whose work it is (anonymous exemplars, a peer's draft, the learner's own).
Criteria Who writes them (given, co-constructed from contrasting exemplars, or left to learners), how many, whether each has a described level and an anchor example, and whether they are simple and objective (format, length, presence of a citation) or subjective and complex (quality of reasoning, audience fit).
Context and activities Setting (classroom, online-instructor-led, online-self-paced, workplace-clinical), single-session or days-weeks, group size and formation, synchronous or asynchronous, whether a teacher reaches each group, how judgments are recorded and revealed, and whether any grade depends on them.
Learner-valued goal Ask what learners want from it: to know what good work looks like, to improve their own draft, to choose a source or option for a real project, a fair grade, or getting it done. Record disagreement with the designer objective.
Designer objective Which capability is the target: judging new work accurately against the criteria, explaining a judgment, improving one's own work, or the discussion skill of settling a disagreement by evidence. Name the one the assessment will weigh.
Outcome An individual judgment of new, unseen objects scored against a reference (assessment-accuracy), on immediate-performance or delayed-retention; the quality of a revised own draft; or an explanation scored on a rubric. The group's consensus rating is not this outcome.

A request to "have students evaluate each other's work" does not specify these. Ask what is evaluated, against which criteria and reference, and how each learner's judgment will be checked before choosing group size, criteria format or the number of rounds.

Sequence and conditional policy

One session (about 45–50 minutes, proposed):

  1. Present the object and the criteria (5 minutes). Use two or three objects that differ in quality on the criteria; in early sessions these are anchor exemplars whose quality the teacher has already judged. Give three or four criteria, each with a described level. Evidence status: rubric guidance for peer raters [+M] helped MOOC writers place samples at the right quality level, mostly on subjective, complex criteria; the earlier page's step 2.
  2. Model one judgment (5 minutes, early sessions only). The teacher thinks aloud through one exemplar: the criterion, the evidence in the object, the level, and what would move it up a level. Evidence status: self-assessment accuracy [~M]: secondary students who studied modelling examples of self-assessment judged their own work more accurately. That was self-assessment, not judging others' work.
  3. Judge alone, in writing (8–10 minutes). Each learner rates each object on each criterion and writes one reason that points to the object. No talk yet. Evidence status: untested here; it is what makes the later comparison informative, and it follows the "answer alone first" step in the peer discussion [+M] studies; the earlier page's step 3.
  4. Reveal and compare in groups of three or four (10–15 minutes). Show all ratings at once before anyone speaks. Discuss only the criteria where ratings differ by a level or more; each side states the evidence in the object; the group records a judgment or a reasoned disagreement. Evidence status: peer assessment [+M] improved academic performance over no assessment across 54 studies, in configurations that included peer dialogue; none of them isolates group comparison of judgments. The earlier page's steps 4–5.
  5. Compare with the reference (5–10 minutes). The teacher gives the reference judgment and its reasons, and the class discusses where groups departed from it and why. Evidence status: in the peer discussion [+M] studies, discussion followed by instructor explanation gave larger gains than either alone, for weak and strong students; those were concept questions in genetics, not judgments of quality.
  6. Check each learner alone (5–10 minutes). Each learner judges one new object, or revises their own draft against the same criteria. Score the judgment against the reference, or the revision on the rubric. Evidence status: rubric-referenced self-assessment [+M] after reviewing a model essay raised middle-school writers' essay scores; the assessor benefit [+M] is that judging peers' work may help the judge's own, a correlation in its one direct study.

Across sessions (four to six sessions over a unit, proposed; untested):

  • Sessions 1–2: teacher-supplied criteria and anchor exemplars of known quality; modelling (step 2) on; groups judge anonymous work, never classmates'.
  • Sessions 3–4: learners co-construct or refine the criteria by contrasting a strong and a weak exemplar; modelling dropped; objects are anonymised peer drafts; the reference comparison (step 5) shrinks to the criteria where groups disagreed with it.
  • Sessions 5–6: learners judge real peer work and then their own drafts; criteria are named but no longer scored line by line where learners already agree with the reference; disagreement is recorded rather than forced to consensus.
  • Move on when a learner's alone judgments of new objects match the reference within one level on most criteria (for example, four of five) in two consecutive sessions. Keep a learner on anchor exemplars, with modelling, while their judgments sit two or more levels from the reference. Both thresholds are proposals to set locally.
Observation Discriminating probe Proposed action, with its uncertainty
The group's judgment matches the reference, but a member's alone judgment of a new object does not Ask that member to justify the group's rating on one criterion from the object; compare their step 3 rating with the group's. If they cannot point to the evidence, the agreement was adopted, not reached: keep the alone judgment and the reveal-before-talk step, and give that member the first turn. If they can justify it but misjudge new work, give more anchor exemplars at the borderline. Neither branch is tested.
Ratings converge quickly on the first or most confident view Compare step 3 ratings with the group's final rating; check who spoke first. If final ratings track one member rather than the evidence, reveal ratings simultaneously, rotate who speaks first, and require one piece of evidence per rating. Agreement is not evidence of accuracy.
Learners agree with each other but not with the reference Ask which criterion the group read differently, and show one anchor at the disputed level. If the criterion is ambiguous, rewrite it with a described level and an anchor. If the group shares a misconception about quality, model a judgment on that criterion. Proposed.
Judgments are generic ("good", "clear") with no evidence from the object Ask for the line, figure or feature that earned the rating. If learners cannot find one, the criterion may be beyond their domain knowledge; narrow to criteria they can see, or give an exemplar pair that differs only on it. Rubric guidance helped most on subjective, complex criteria, so this is where it is most worth adding.
Learners rate their peers' work accurately, but not their own Have them rate their own draft, then rate it again as if it were anonymous, against the same anchors. If the anonymous rating is closer to the reference, the gap may be self-favouring rather than criterion knowledge; keep judging anonymous work first and then their own. If both are far off, the criterion knowledge is missing. Untested.
A learner with relevant expertise is overruled by the majority Ask the group to state the evidence for each side, and compare with the reference. Record the dissent with its evidence; do not require consensus where the reference sides with the minority. The earlier page warned against consensus erasing minority interpretations.

Fitting the design to a situation

The three facts that most change the decision:

  • Whether learners know the domain well enough to see quality. Learners with weaker command of material judge their own performance least accurately, and accuracy rises with familiarity and domain specificity. If unstated, ask: given one strong and one weak example, can they say which is stronger and why, without help?
  • Whether there is a defensible reference judgment. Without one, the group can only agree with itself, and the step that corrects shared misjudgments disappears. If unstated, ask: who has judged these objects already, and would informed judges agree?
  • What the judgments are for: learning, or a grade. Peer assessment's effects are on learning when it is used formatively; rubric use has come with more stress and a performance-avoidance orientation. If unstated, ask: will any learner's mark depend on a peer's or a group's rating?
If the brief says… Then change… Basis
Learners are novice in the domain (cannot yet tell strong from weak work unaided) Use anchor exemplars with known quality, three or four criteria with described levels, teacher modelling each session, and the reference comparison after every round; keep criteria to what learners can see in the object. Delay judging classmates' work until alone judgments of anchors are within a level. Prior knowledge and self-assessment [~M] and self-assessment accuracy [~M], both about judging one's own performance, carried to judging others'
Learners are advanced or practising professionals Have them co-construct criteria, judge genuinely borderline cases, and record reasoned disagreement instead of forcing consensus; use a rubric for the shared vocabulary, not as a score sheet. Rubric guidance [~M]: in a graduate course a rubric changed feedback content but restrained reflective and divergent commentary
Learners are child (under 12) Use two objects at a time, two or three criteria phrased as questions with picture or worked anchors, and a short spoken comparison in pairs; the teacher gives the reference each time. Peer assessment [+M]: the meta-analysis includes primary students and found no reliable difference by education level; the configuration is untested
The criteria are subjective or complex (quality of argument, audience fit), not simple and objective Train raters on graded samples at each level before they judge; spend discussion time on these criteria and skip discussion of simple ones. Rubric guidance [+M]: MOOC writing raters, benefit concentrated on subjective, complex criteria and negligible on simple ones
The goal is better own work, not only better judgment End every session with the learner revising their own draft against the same criteria, after judging others' work; score the revision. Rubric-referenced self-assessment [+M] (middle-school writers; pre-service teachers); assessor benefit [+M], correlational in its direct study
The objects are online sources (credibility of websites, claims) Make leaving the site to check who is behind it and what others say (lateral reading) the criterion, rather than a checklist of on-page features; have the group compare what each member found off the page. Lateral-reading instruction [+M] (high-school and university students); one-session checklist instruction [~M] did no better than the alternatives it was compared with
online-instructor-led or asynchronous Collect each rating in a form that locks before others' ratings are visible; reveal all together; hold the discussion in a thread limited to criteria where ratings differ by a level or more; post the reference after the thread closes. Peer assessment [+M] found no difference between online and offline delivery; the assessor-benefit meta-analysis [+M] found computer-mediated peer assessment did better than paper. The locking and thread rules are untested
A large class, or no teacher reaches each group Replace group consensus with the mean of four or five independent ratings per object, and give the reference to the whole class afterwards; discuss only the objects where the mean departs most from it. Peer judgment accuracy [~M]: with a well-specified rubric, averaged ratings by AP English students correlated with expert scores about as well as teachers' did. Correlational, one subject
High stakes: peer or group ratings would count towards marks Keep collaborative evaluation formative; grade only individual judgments of new objects against the reference, or the revised own work by the teacher; make peer work anonymous. Rubric use [~M] raised stress and a performance-avoidance orientation in pre-service teachers; anonymous sharing [~W] is the authors' account, untested
Status differences or dominant voices in groups Private written ratings first, simultaneous reveal, rotate who speaks first, and allow anonymous submission of ratings to the teacher. Anonymous sharing [~W], authors' account of one undergraduate physics course, no test
single-session, no follow-up One round: two anchors of known quality, alone judgment, reveal and compare, reference, then one new object judged alone. Do not attempt co-constructed criteria. Untested proposal

Seven rows rest on a claim (novice learners, advanced learners, subjective criteria, own-work revision, online sources, online delivery, large classes) and three more lean on a weak or carried claim (children, high stakes, status); each Basis cell says when the claim's learners, outcome or setting differ from the row's. The single-session row is an untested proposal.

Choosing configurations from evidence

  • Peer assessment against none. Peer assessment improves academic performance by a small average amount relative to no assessment or to teacher assessment [+M]: a meta-analysis of 54 control-group studies (k = 141) across primary, secondary and tertiary students found peer assessment (grading, peer dialogue, written feedback) improved academic performance over no assessment (g = 0.31) and over teacher assessment (g = 0.28), and did not differ significantly from self-assessment (g = 0.23); online or offline delivery, frequency and education level did not change it. It supports having learners judge work against criteria. It does not compare group consensus with individual judging, and its outcome is academic performance, not judgment accuracy.
  • The judge may gain. The quality of feedback students give peers predicts gains in their own work, but no recorded study isolates a causal benefit of the assessor role [+M]: a meta-analysis of 58 studies found participation in peer assessment, in either role, associated with a 0.291 SD gain, larger with rater training and with computer-mediated delivery; in 43 undergraduate teacher-education students, the quality of feedback a student gave predicted their own final project, controlling for its initial quality, while the feedback received did not. The pooled estimate mixes roles and the direct study is correlational, so carry it to "judging others' work is worth including", not to "judging causes better own work".
  • Rubric guidance changes what raters see. In two online comparisons, a rubric changed what peer reviewers commented on (one study) and helped peer raters place writing in quality tiers, most for complex criteria (the other); the first study says the rubric restrained reflective thinking [+M]: two quasi-experiments (q2, no effect sizes). In a MOOC writing course, guided-rubric training helped raters place samples in novice, intermediate and advanced tiers, mostly on subjective, complex criteria; in a graduate course of 47, a rubric changed feedback content and restrained reflective and divergent commentary. It supports rater training on graded samples; it also warns that a rubric can narrow discussion.
  • Structure is not shown to raise learning. Structured criteria in peer assessment [~M]: in the same 54-study meta-analysis, studies giving reviewers a script or rubric (g = 0.29) did not differ significantly from freeform peer assessment (g = 0.42); the freeform subgroup was small (13 studies). This limits the model: criteria may make judgments more consistent without making the activity more effective for learning, and the evidence here does not separate the two.
  • Accuracy needs knowledge, or training, or averaging. Self Assessment Accuracy Is Low Without Training [~M]: in a randomized experiment with 80 Dutch secondary students, those who studied modelling examples of self-assessment judged their own genetics problem-solving more accurately (ηp² = .10); across 51 higher-education studies, self–teacher agreement was inconsistent, closer in advanced courses. Prior Knowledge Needed For Accurate Self Assessment [~M]: across 22 meta-analyses self–performance agreement was moderate (mean r = .29), higher for domain-specific judgments and familiar tasks, and the lowest-performing undergraduates were the most overconfident. In two secondary-school writing studies, peer reviewers' own performance predicted how helpful their feedback was but not how accurately it identified problems [~M]: in 234 high-school writers, assessors' own writing performance was unrelated to whether they identified real problems but predicted how helpful their comments were; with a well-specified rubric, 1,215 AP English students' averaged ratings correlated with expert scores at roughly .4–.7. All of it concerns individual judgment, own or of peers; none tests whether group discussion improves it. It is the reason for the reference comparison and the alone check.
  • Discussion, then explanation. Peer Discussion Improves Conceptual Understanding [+M]: in undergraduate genetics, discussion of a concept question improved performance on an isomorphic question answered alone, even in groups where no one had first known the answer; discussion followed by instructor explanation gave larger gains than either alone, for weak, medium and strong students. These were concept questions with a right answer, not judgments of quality; carry it to the order "judge alone, discuss, then reference", no further.
  • What the criteria are matters as much as having them. Civic Online Reasoning Instruction Improves Evaluation [+M] and One-session checklist instruction [~M]: lateral-reading instruction improved high-school and university students' judgments of online credibility over matched or randomised controls, while a one-session CRAAP checklist did no better than the methods it was compared with. For source evaluation, the criteria themselves carry much of the effect.

Do not rank collaborative evaluation against individual peer review, self-assessment or teacher-led marking from these claims: none compares them, and none measures the time a group discussion costs. Do not read a group's consensus, or its agreement with the reference, as each member's judgment. A precise abstention names the missing comparison (group judgment with a reference comparison against individual judgment with the same reference, on the accuracy of alone judgments of new work at a stated horizon) and the local observation that would stand in for it.

The evidence by question

Further evidence, not yet read against this model

Claims the earlier page cited. Each keeps its old direction, capped by the claim's recorded evidence. None has been re-read for the model above; treat them as candidates for it, not as part of it.

Illustrative design instance and observation record

Invented illustration, not a tested lesson. First-year undergraduates (adult, postsecondary) in a research-methods course must choose sources for a group project; they are new to judging research (novice for this task). The table changed the default in three ways: the criteria are lateral-reading moves (who is behind this, what do others say about it, what does the cited evidence actually show) rather than an on-page checklist; the first two sessions use four anchor sources the tutor has already judged; and no rating counts towards the mark. In session 1 the tutor models one judgment, each student rates two anchors alone in a form that locks, the group of four reveals and discusses the criteria where ratings differ, and the tutor gives the reference. In sessions 3–4 groups judge sources members have proposed for the project; in session 5 each student judges two new sources alone and rates their own annotated bibliography. One student says she wants "to stop wasting time on bad sources"; the designer's objective is accurate judgment of new sources. They agree to score both the accuracy of the alone judgments and the share of the final bibliography that meets the criteria. The lateral-reading claims cover high-school and university students on credibility of web content; the group comparison and the thresholds are local choices to observe, not settled ones.

Record: object and its reference judgment → criteria as given → each learner's alone rating and reason, per criterion → ratings revealed → which criteria were discussed, who spoke first, what evidence was cited → group judgment or recorded disagreement → departure from the reference and the class's account of it → each learner's alone judgment of a new object, or revision of own work, scored → candidate interpretations (criterion not understood, domain knowledge missing, followed a confident peer, self-favouring on own work) and their basis → reobservation at the stated horizon. Preserve the learner's stated purpose and any change in it. An agent may summarize this record but must not fabricate missing inputs or treat a group rating as any member's capability.

Elements and limits

Assessment, Consensus Discussion, Peer Discussion, Formative Assessment, rubrics and exemplars. Principles the pattern draws on: Formative Assessment, Assessment for Learning, Multiple Methods of Assessment, Peer Feedback/Peer Review and Evaluating Sources.

This pattern is scoped to learners judging the quality of objects against criteria, with a reference judgment available. Where no reference exists and the question is which position to hold, use structured academic controversy; where the aim is giving feedback on an individual's draft, structured peer review. The policy supports observation and design reasoning; whether judging together and comparing with a reference improves each learner's later alone judgments more than judging alone with the same reference, for which learners, at what time cost and at what horizon, remains to be tested.

Claims

Examples

  • Teams comparing possible sources for a research project against credibility criteria.
  • Learners evaluating project proposals for feasibility and impact.
  • Small groups scoring sample responses against a shared rubric and discussing differences.
  • Calibrated Peer Review — learners rate calibration samples of known quality before rating peers' work.

Key Sources

  • McAndrew, P., Goodyear, P., & Dalziel, J. (2006). Patterns, designs and activities: Unifying descriptions of learning structures. International Journal of Learning Technology, 2(2-3), 216-242. https://doi.org/10.1504/IJLT.2006.010632
  • Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. https://doi.org/10.1007/BF00117714
Page metadata
Field Value
type pattern
id collaborative-evaluation
title Collaborative Evaluation
description Learners who judge work against shared criteria alone, compare their judgments in a small group and then with a reference judgment are expected to judge new work more accurately and to improve their own work, where they know the domain well enough to see quality and the ratings carry no grade; no claim tests collaborative evaluation itself, and the evidence is peer-assessment, rubric-training and judgment-accuracy studies carried to it.
status review
grain_size lesson
author McAndrew, Goodyear, and Dalziel
canonical True
generated by claude/unspecified on 2026-10-05
trust unverified — no one has confirmed this page's content
sources 2 — mcandrew-2006, sadler-1989

Frontmatter is the page's machine-readable record — see the Schema & Guide.