The test's exercises were validated by six expert judges who reached practical agreement after modification¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 1918 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Six judges chosen for mental acumen and long study of scientific method independently judged whether each of forty second-edition exercises tested its labelled ability, and exercises were modified until the judges reached practical agreement. → Herring 1918

Evidence¶

Herring 1918¶

Herring, J. P. (1918). Measurement of some abilities in scientific thinking. The Journal of Educational Psychology, IX(10), 535-558. https://doi.org/10.1037/h0075736

q2 · i? · design · r2

Validity study of the second edition's forty exercises: six judges (philosophy, history, mathematics, engineering) worked separately, spending from two to five hours each, and "At last the judges came to a practical agreement" after modification. No effect size is printed.

"Each stated which, if any, of the abilities was tested by each exercise. Not even at first was there a discouraging amount of disagreement. Some exercises were discarded, some modified slightly, and some rebuilt. At last the judges came to a practical agreement."

Discussion¶

Related Claims¶

• Five judges classifying 63 test units reached average inter-judge correlation of 0.542 ± 0.019 on the two scales — related
• Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement — related
• LLM-as-judge automated evaluation can achieve human-level agreement when carefully validated — a broader claim this one bears on
• The second edition was tried with 140 grade and 141 high school pupils; seven exercises were discarded, and about 1800 grade pupils took the final test in May 1917 — related