Defining and quantifying reliability and measurement error is particularly challenging in epistemic games for three principal reasons¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q1
Currency · newest study 2010 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2013: tool and model accuracy changes quickly

Subclaims¶

q1 i? Reliability and measurement error in epistemic games are hard to define and estimate because complex tasks induce dependencies across observations, multiple layers of human judgment generate indicators, and rich process/product data may be distal to the desired interpretations. → Rupp 2010

Evidence¶

Rupp 2010¶

Rupp, A.A., Gushta, M., Mislevy, R.J., & Shaffer, D.W. (2010). Evidence-centered Design of Epistemic Games: Measurement Principles for Complex Learning Environments. Journal of Technology, Learning, and Assessment, 8(4). https://ejournals.bc.edu/index.php/jtla/article/view/1623

q1 · i? · theoretical · r3

Theoretical argument in the paper's reliability section. The authors argue that "the complexity of the tasks, which induce dependencies across observations" undermines traditional error minimization, and that resulting statistics may even turn out to be game-specific.

"Defining and quantifying reliability and measurement error of scores is particularly challenging in the context of epistemic games for three principal reasons. First, it is very challenging to minimize measurement error from an assessment design perspective due to the complexity of the tasks, which induce dependencies across observations."

Studies bearing on this claim¶

Can what a learner has learned in an epistemic game be measured reliably from the record of play? This table gathers the studies behind this claim and the 5 narrower claims that bear on it: 2 distinct studies (1 theoretical, 1 causal), q1–q2, 0 of 2 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Argumentation quality is associated with denser co-occurrence of scientific reasoning and self-regulation processes — related
• Multidimensional latent variable models rarely support more than about four to six latent variables, limiting fine-grained SKIVE modeling — related
• Developers of the principal practice instruments provided only partial information about the instruments' reliability and validity — a narrower finding that bears on this claim
• Organizing performance measures into outcomes, impacts, and processes provides rich, fair, and diagnostically useful information — related
• Student growth modeling and teacher value-added modeling are complex due to non-random assignment, measurement error, multidimensionality, and covariates — related
• Scovel argued anxiety is not a simple, unitary construct quantifiable into high or low amounts — related
• Segmentation boundary choices differentially affect statistics computed on epistemic-game process data — a narrower finding that bears on this claim
• The WD statistic is more useful for differentiating between simulated learner types when games are played longer — related
• Easy, highly specific simulated tasks compress the range of the WD statistic across learner types, while well-designed tasks widen it — related
• In simulated epistemic games, the weighted density statistic distinguishes simulated learner types with distinct mastery trajectories, with the expert trajectory showing the largest WD values — related
• Measurement models based on vaguely quantified behavioral frequency reports differ in structure from those based on numerically quantified reports — related
• Formal education exhibits the defining properties of a complex system — related