Improve international assessments for evaluating instructional practices via panel designs, student-questionnaire practice items in every wave, more refined items, and no unsupported quality scales¶

Strategy · All strategies
Evidence · 6 claims (6 for) · 1 study (1 associational), q2 · 1 of 1 report an effect size · 6 claims rest on one study

Description¶

The article proposes four concrete improvements to make TIMSS more informative for evaluating instructional practices. These are: strengthening the design "so that the same students are tested more than once", including measures of instructional practices in the international student questionnaire in every wave, refining the under-specified items so they tap specific aspects such as cognitive activation, and not providing instructional-quality scales for which there is no evidence that each item represents quality teaching.

Design Implications¶

Context¶

Requirements¶

• Measures of instructional practices must be included in the international student questionnaire for cross-country examination of relations to achievement.
• Refined items are needed because assessing the actual quality of instructional activities requires more refined items than current TIMSS questionnaires.

Constraints¶

• TIMSS data are anonymized, so linking to prior achievement data is generally not possible except in special cases like Swedish TIMSS 2015.

Target Learners¶

• eighth-grade mathematics students

Target Learning Goals¶

• mathematics achievement
• evaluation of instructional practices

Claims¶

• Country differences in reported levels of use of instructional practices were highly consistent across TIMSS waves (ClassListen r = .70; ClassDaily r = .91) [+W]
• The IEA-provided Instruction to Engage Students in Learning Scale has practically zero correlation to student achievement, according to IEA (2011) exhibit data cited in the article [+W]
• The three instruction items intercorrelate strongly enough to form an internally consistent scale, yet achievement is positively associated with two items and negatively with the third, so their common variance does not represent instructional quality [+W]
• Across countries in TIMSS 2007, the strongest estimated effects on math achievement were a positive effect of ClassMemo followed by a negative effect of ClassDaily, with median effects matching the Swedish pattern [+W]
• In Swedish TIMSS 2015 with grade-6 pretest controls, a unit increase in ClassMemo was associated with 0.27 standard deviations improvement in relative math achievement, while ClassListen and ClassDaily effects were non-significant [+W]
• Class-level student reports of memorizing formulas and listening to lectures positively predict mathematics achievement, while relating math to daily life negatively predicts it, across Swedish TIMSS waves 2003–2015 [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Kimmo Eriksson, Ola Helenius, Andreas Ryve. (2019). Using TIMSS items to evaluate the effectiveness of different instructional practices. Instructional Science. https://doi.org/10.1007/s11251-018-9473-1