Log-likelihood comparisons indicate Pychometrik can produce more accurate item parameter estimates than the existing tool for items with RIT differences larger than 2¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? For the 1,180 AGC items, the mean log-likelihood was -2803.94 for the existing tool versus -2801.66 for Pychometrik; for the 517 items with RIT differences the means were -2739.59 versus -2734.39. → He 2021

Evidence¶

He 2021¶

He, W., Bo, E., Meyer, P., & Grandgeorge, R. (2021). A comparison of item parameter estimates in Pychometrik and the existing item calibration tool. NWEA. https://www.nwea.org/research/publication/a-comparison-of-item-parameter-estimates-in-pychometrik-and-the-existing-item-calibration-tool/

q2 · i? · design · r2

Diagnostic analysis comparing log-likelihood values computed on item RITs from the two systems, described via Newton-Raphson and independent replication code; "on average, Pychometrik provides better item parameter estimates than the existing tool".

"Table 3.4 presents the average log-likelihood values for all 1,180 AGC items based on the item RITs and the 517 items with RIT differences from the two systems. The results suggest that, on average, Pychometrik provides better item parameter estimates than the existing tool."

Discussion¶

Related Claims¶

• Item calibration results from Pychometrik and the existing tool are mostly comparable across 1,180 AGC-calibrated MAP Growth items, with average RIT differences ranging from 0.33 (mathematics) to 0.91 (science) — related
• In a fixed-person calibration simulation with 60 Rasch items and 1,200 simulees, estimates from both tools were comparable to true parameters, with the existing tool underestimating difficulty by 0.03 RIT on average — related