Maintain reviewer quality through explicit benchmark checks and implicit analytics on scoring drift¶

Strategy · All strategies
Evidence · no claims cited

Description¶

Step 5 of the workflow is an ongoing process with two strands. In explicit maintenance, reviewers are periodically assigned benchmark samples to score, or paired with additional reviewers to check for inter-rater reliability, invisibly to the reviewer. Implicit maintenance uses system analytics, such as whether a reviewer tends to score higher or lower than others, flagging reviewers who apply stricter or more generous scores until improved consistency with the Benchmarks Database is demonstrated.

Design Implications¶

Context¶

Requirements¶

• A system that captures reviewer score agreement data over time
• Periodic assignment of benchmark samples or paired reviews

Constraints¶

• The article notes trends in score outcomes naturally shift over time, so monitoring must be ongoing

Target Learners¶

• certified micro-credential reviewers

Target Learning Goals¶

• sustained inter-rater reliability and scoring consistency over time

Related Strategies¶

• Separate feedback provision from competency certification, using feedback-approved reviewers or pre-submission review

Examples¶

-

Key Sources¶

• Michelle Riconscente, Designs for Learning, Inc. (2016). Designing a Process for Micro-credentials at Scale. Digital Promise. https://digitalpromise.dspacedirect.org/items/ddd022c9-a271-4548-9667-38dbfe9bf9c6