Consider differential test effort when interpreting achievement gap estimates among student subgroups¶

Strategy · All strategies
Evidence · 5 claims (4 for, 1 mixed) · 1 study (1 associational), q3 · 0 of 1 report an effect size · 5 claims rest on one study

Description¶

This strategy follows from the brief's central concern: because "test effort differs substantially across student gender and racial subgroups," achievement gap estimates drawn from low-stakes or standardized test data may reflect differences in engagement rather than true achievement differences. Educators and analysts using gap estimates as a policy metric should therefore ask whether observed subgroup differences could be contaminated by differential test effort before acting on them. The brief treats achievement gaps as "an important policy metric" whose interpretation is at stake.

Design Implications¶

Context¶

Requirements¶

• Access to information or indicators about student test effort on the assessments used to compute gap estimates

Constraints¶

• The supplied text does not print a method for measuring or adjusting for test effort; it only establishes that effort differs across subgroups

Target Learners¶

• K-12 students assessed with standardized achievement tests

Target Learning Goals¶

• Accurate interpretation of achievement gap estimates across gender and racial subgroups

Claims¶

• Differential student test effort may bias achievement gap estimates [+W]
• Student test effort differs substantially across student gender and racial subgroups [+W]
• Test effort differs substantially by student subgroup, with males rapidly guessing nearly twice as often as females in later grades and Black students rapidly guessing more often than White students [+W]
• Basic conclusions about male-female achievement gaps and changes in gaps as students progress through school may change when models account for test effort [+W]
• Differences in rapid guessing between subgroups generally do not impact substantive interpretations of achievement gaps [~W]

Related Strategies¶

• Investigate why subgroup differences in test effort exist to support educators

Examples¶

-

Key Sources¶

• James Soland. (2019). Are achievement gap estimates biased by differential student test effort? NWEA Research. https://www.nwea.org/research/publication/are-achievement-gap-estimates-biased-by-differential-student-test-effort/