Survey items show good discrimination overall, but several items exhibit ceiling effects that limit measuring change over time¶

Claim · All claims
Evidence · 1 study (2 entries) · design r3 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly

Subclaims¶

q2 i? All but three of the 32 items had item-total correlations greater than 0.40, indicating good discrimination. → Kao 2023 q2 i? Items with mean scores greater than 4 have ceiling effects, making them less than ideal for measuring changes over time. → Kao 2023 (2)

Evidence¶

Kao 2023¶

Kao, Y., Murphy, D., Hubbard Cheuoua, A., Kannan, P., Tsan, J., Jennings, K. E., Smith, H., Emanuel, S., & Miller, E. R. (2023). Survey development for assessing student computing career intentions: Technical report. WestEd. https://eric.ed.gov/?id=ED638631

q2 · i? · design · r3

Classical item analysis in the large-scale field test computed item-total correlations for all 32 items. The article reports "all but three items on the survey had item–total correlations greater than 0.40", meeting the threshold for good discrimination.

"It can be seen from Tables 11 through 17 that all but three items on the survey had item–total correlations greater than 0.40, and therefore have good discrimination properties."

Kao 2023 (2)¶

Kao, Y., Murphy, D., Hubbard Cheuoua, A., Kannan, P., Tsan, J., Jennings, K. E., Smith, H., Emanuel, S., & Miller, E. R. (2023). Survey development for assessing student computing career intentions: Technical report. WestEd. https://eric.ed.gov/?id=ED638631

q2 · i? · design · r3

The same item analysis found all item means exceeded 3, with some above 4. The article states items with means greater than 4 "can be considered to have ceiling effects", limiting sensitivity to change over time.

"The item means shown in Tables 11 through 17 are all greater than 3, and in some cases greater than 4, indicating that participants tended to agree or strongly agree with the items. Items with mean scores greater than 4 can be considered to have ceiling effects, making them less than ideal for a survey aiming to measure changes over time."

Discussion¶

Related Claims¶

• The full 32-item computing career intentions survey shows very good internal consistency (alpha = 0.94) in a large-scale field test — related
• Correlations between estimated and true scores diminish over time mainly with easy items, especially with a true slope of .5 and only four items, likely due to ceiling effects — related
• Evidence for collaboration and communication growth was positive but limited by ceiling effects, while design-thinking growth was mixed — related
• Negative affective experience items were dropped due to floor effects and poor discrimination — related
• Polychotomously scored IF-AT physics exam items show excellent discrimination, with a mean item-total correlation of r′ = 0.41 ± 0.13 across 45 items — related
• Independent LLM evaluators agreed on agent rankings but differed markedly in leniency, with ceiling effects compressing between-agent discrimination — related
• Several of the group tests (Discriminating Colors, Mutilated Pictures, Tying Bowknot) showed little discriminative capacity as group tests and are slated for substitution — related