In a fixed-person calibration simulation with 60 Rasch items and 1,200 simulees, estimates from both tools were comparable to true parameters, with the existing tool underestimating difficulty by 0.03 RIT on average¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Both tools' estimates were close to true parameters; Pychometrik's standard deviation (9.47) was slightly closer to the true SD (9.46) than the existing tool's (9.51), and the existing tool's bias was -0.03 RIT. → He 2021
Evidence¶
He 2021¶
He, W., Bo, E., Meyer, P., & Grandgeorge, R. (2021). A comparison of item parameter estimates in Pychometrik and the existing item calibration tool. NWEA. https://www.nwea.org/research/publication/a-comparison-of-item-parameter-estimates-in-pychometrik-and-the-existing-item-calibration-tool/
q2 · i? · design · r2
Simulation study in which 60 Rasch items were generated from the standard normal distribution and administered to 1,200 simulees; bias, MAD, and MSE were used to compare both tools' estimates with each other and with true parameters.
"In general, the item parameter estimates from both tools are comparable to the true item parameters, but those from Pychometrik are slightly closer to the true parameters than those from the existing tool with a smaller standard deviation (i.e., 9.47 (Pychometrik) vs. 9.46 (true) vs. 9.51 (existing tool))."
Discussion¶
Related Claims¶
- When item difficulty and simulee ability ranges align (minimum -10, maximum 8), Pychometrik recovery is accurate, with correlations above 0.99 across all three sample sizes; recovery degrades above the simulee ability maximum of 7.94 — related
- Item calibration results from Pychometrik and the existing tool are mostly comparable across 1,180 AGC-calibrated MAP Growth items, with average RIT differences ranging from 0.33 (mathematics) to 0.91 (science) — related
- Log-likelihood comparisons indicate Pychometrik can produce more accurate item parameter estimates than the existing tool for items with RIT differences larger than 2 — related
- In the Pychometrik item parameter recovery simulation across 27 conditions (Grades K-8, sample sizes 500, 1,000, and 2,000), recovery accuracy increased with sample size and was best for Grade 8 simulees — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
simulation-comparability-60-items-1200-simulees |
title |
In a fixed-person calibration simulation with 60 Rasch items and 1,200 simulees, estimates from both tools were comparable to true parameters, with the existing tool underestimating difficulty by 0.03 RIT on average |
description |
In a fixed-person calibration simulation with 60 Rasch items and 1,200 simulees, estimates from both tools were comparable to true parameters, with the existing tool underestimating difficulty by 0.03 RIT on average |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2021 |
recheck_by |
2024 |
generated |
by process:wiki-ingest on 2026-10-08 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — he-2021 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.