Segmentation boundary choices differentially affect statistics computed on epistemic-game process data¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2010 · not yet checked against its sources · not reviewed by a person
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? Initial research showed that different classes of statistics are differentially sensitive to segmentation boundary shifts: absolute values of statistics for individual SKIVE elements are more strongly affected than relative weights of the SKIVE elements, which appear reasonably robust. → Rupp 2010

Evidence¶

Rupp 2010¶

Rupp, A.A., Gushta, M., Mislevy, R.J., & Shaffer, D.W. (2010). Evidence-centered Design of Epistemic Games: Measurement Principles for Complex Learning Environments. Journal of Technology, Learning, and Assessment, 8(4). https://ejournals.bc.edu/index.php/jtla/article/view/1623

q2 · i? · theoretical · r3

The paper reports, citing Rupp et al. (2009b), initial research on process-data segmentation in epistemic games. It found statistics' absolute values more affected by boundary shifts than relative weights of SKIVE elements, which appeared reasonably robust. The cited study's design is not described in the paper.

"Initial research in this area has shown that these segmentation boundaries are important and that different classes of statistics are differentially sensitive to this segmentation. For example, the absolute value of certain statistics for individual SKIVE elements are more strongly affected than the relative weights of the SKIVE elements, which appears to be reasonably robust (e.g., Rupp et al., 2009b)."

Discussion¶

Related Claims¶

• Defining and quantifying reliability and measurement error is particularly challenging in epistemic games for three principal reasons — the general claim this finding bears on, with the studies behind it
• In simulated epistemic games, the weighted density statistic distinguishes simulated learner types with distinct mastery trajectories, with the expert trajectory showing the largest WD values — related
• The WD statistic is more useful for differentiating between simulated learner types when games are played longer — related
• Easy, highly specific simulated tasks compress the range of the WD statistic across learner types, while well-designed tasks widen it — related
• In pairwise WD analyses, learner trajectory similarity dominates variation in percentage-overlap values (57.90% of variation) while remaining design factors are essentially zero — related
• In the simulation, task complexity and task difficulty explain the majority of variance in the individual-learner WD statistic, with some effect of task specificity — related
• A sensitivity analysis indicates the content-derived information density metrics are robust to reasonable operationalizations — related
• In case examples, leaders and team members showed markedly different average peaks-per-segment EDA values during the same simulation segments — related
• No published research describes colleges using segmentation to improve student success, though authors and experts see promise in the approach — related