The PS method shows systematic overestimation of the differentiation component that exceeds the 10% bias threshold in small and medium level-1 samples¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Simulation study 1 found systematic overestimation of the differentiation component for the PS method, exceeding the acceptable 10% relative-bias threshold in small and medium level-1 conditions, whereas the ML-LVM bias stayed below the threshold. → Julian F. Lohmann 2025

Evidence¶

Julian F. Lohmann 2025¶

Julian F. Lohmann, Nils Machts, Jens Möller, Steffen Zitzmann. (2025). A More Comprehensive, More Reliable Multilevel Approach for Assessing and Modeling Teacher Judgment Accuracy Using Latent Variables. Educational Psychology Review. https://doi.org/10.1007/s10648-025-10029-z

q2 · i? · design · r2

Simulation study 1, population-level results (Table 1) across nine sample-size conditions with 250 datasets each, evaluated percentage bias against the Muthén and Muthén 10% threshold. The article reports the PS overestimation "exceeded the acceptable threshold of 10%" while individual rank and level bias was negligible for both methods.

"Regarding bias, a systematic overestimation was observed for the differentiation component across conditions for both approaches. However, overestimation was much higher for the differentiation component when using the PS method. Bias even exceeded the acceptable threshold of 10% in the small and medium level-1 sample size conditions."

Discussion¶

Related Claims¶

• The ML-LVM yields higher true-positive rates for detecting moderation effects of judgment accuracy components than the PS method across most conditions — related
• The ML-LVM provides more accurate individual-level estimates of judgment accuracy components than the person-score (PS) two-step method — related
• The ML-LVM yields lower parameter bias and higher coverage rates for regression weights predicting learning outcomes than the PS two-step method — related