Facet difficulty and intercorrelation patterns: analyzing givens was easiest, evaluating progress hardest, and planning correlated 0.98 with the overall estimate¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r1–r2 · q3 · i3 large
Currency · newest study 2016 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2019: tool and model accuracy changes quickly

Subclaims¶

q3 i? The analyzing givens and constraints facet was negatively skewed with a relatively high mean score, suggesting it was the easiest facet, while the evaluating progress facet and MicroDYN measures were positively skewed, indicating relative difficulty. → Shute 2016 q3 i3 The correlation between the overall problem solving estimate and the planning a solution facet was r = 0.98, almost indicating identity. → Shute 2016 (2)

Evidence¶

Shute 2016¶

Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106-117. https://doi.org/10.1016/j.chb.2016.05.047

q3 · i? · associational · r2

Descriptive analysis of the validation study measures on a 0-1 scale. The article reports the analyzing givens facet was negatively skewed with a high mean, suggesting it "was the easiest of the four problem solving facets", while evaluating progress and the MicroDYN measures were positively skewed and relatively difficult.

"The distribution of the analyzing givens and constraints facet was negatively skewed. This, coupled with its relatively high mean score, suggests that it was the easiest of the four problem solving facets for the students."

Shute 2016 (2)¶

Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106-117. https://doi.org/10.1016/j.chb.2016.05.047

q3 · i3 · associational · r1

Correlational analysis among the in-game estimates in the validation study. The article reports the overall problem solving estimate correlated r = 0.98 with the planning a solution facet, and attributes this to planning being conceptually the broadest facet, while noting regression results still differ across facets.

"What is a little surprising is the correlation between the overall Problem Solving estimate and the Planning a Solution facet ( r ¼ 0.98), almost indicating identity."

Discussion¶

Related Claims¶

• Adding MicroDYN rule application significantly improves prediction of overall problem solving, planning, tool usage, and evaluating progress, but not analyzing givens — related
• MicroDYN is a stronger predictor of the in-game problem solving estimates than Raven's, with the rule-application facet more relevant than rule acquisition — related
• Stealth assessment facets show good internal consistency (Cronbach's alpha = 0.76) while MicroDYN reliabilities were low in this middle-school sample — related
• Problem isomorphs with identical structure differ greatly in difficulty; Monster Change took 16 times as long to solve as the standard problem — related
• Zoo U subscales are intercorrelated in patterns the authors attribute to real-world overlap among social skills competencies — related