ECD framework as a conceptual and rhetorical structure for designing simulation studies in games-based assessment¶

Theory · All theories
Evidence · 5 claims (5 for) · 1 study (1 design), q2 · 1 of 1 report an effect size · 5 claims rest on one study

Description¶

The article extends the evidence-centered design (ECD) framework beyond assessment design to the design of simulation studies investigating statistical methods for games-based assessment. Its conceptual assessment framework components—student model (SKIVE variables), task model (evidentiary boundaries), evidence model (evidence identification and accumulation), and assembly model (design matrices)—organized the simulation's learner, task, and game conditions. The authors report that ECD provided "a much -needed conceptual and rhetorical structure that team members could use to effectively communicate with one another" across methodological traditions.

Design Implications¶

Context¶

Requirements¶

• An interdisciplinary design team needing shared structure for articulating the core drivers of their evidentiary assessment argument: characteristics of learners, tasks, and their associated interactions
• A defensible mapping of simulation parameters onto features of the real-life epistemic games

Constraints¶

• In early-stage development, only some of the design features of the game were stable enough to be operationalized as simulation design conditions
• Games-based assessment is characterized as a relatively ill-defined methodological domain

Target Learners¶

• learners in epistemic games (simulated learner types in this study)
• interdisciplinary game development and assessment research teams

Target Learning Objectives¶

• detecting learner differences in domain-specific epistemic competencies
• evaluating the performance of statistics used for characterizing learner performance

Claims¶

• Easy, highly specific simulated tasks compress the range of the WD statistic across learner types, while well-designed tasks widen it [+M]
• The WD statistic is more useful for differentiating between simulated learner types when games are played longer [+W]
• In the simulation, task complexity and task difficulty explain the majority of variance in the individual-learner WD statistic, with some effect of task specificity [+W]
• In pairwise WD analyses, learner trajectory similarity dominates variation in percentage-overlap values (57.90% of variation) while remaining design factors are essentially zero [+M]
• In simulated epistemic games, the weighted density statistic distinguishes simulated learner types with distinct mastery trajectories, with the expert trajectory showing the largest WD values [+M]

Related Theories¶

• Evidence-Centered Design framework structures assessment design through student, task, evidence, assembly, and presentation models

Examples¶

-

Key Sources¶

• Sweet, S. J., & Rupp, A. A. (2012). Using the ECD Framework to Support Evidentiary Reasoning in the Context of a Simulation Study for Detecting Learner Differences in Epistemic Games. Journal of Educational Data Mining, Special Issue, Article 5, Volume 4. https://jedm.educationaldatamining.org/index.php/JEDM. doi:10.5281/zenodo.3554649