Taxonomy of item difficulty prediction approaches: expert-driven versus machine-driven

Research Method · All research methods
Evidence · 2 claims (2 for) · 2 studies (1 associational, 1 design), q2 · 1 of 2 report an effect size · 2 claims rest on one study

Description

The review introduces a classification (its Fig. 1) that distinguishes two families of item difficulty prediction approaches. Expert-driven approaches are qualitative, resting on "an expert consensus that exploits their domain knowledge and experience" and on pre-defined features from educational taxonomies or experts' difficulty perceptions. Machine-driven approaches quantify difficulty using statistical or data-driven prediction models to enhance scalability and minimise human intervention. The review uses this taxonomy to organise the field and to motivate its focus on automatic, text-based, a priori prediction.

Accounts

• Taxonomy of item difficulty prediction approaches: expert-driven versus machine-driven: The review introduces a classification (its Fig. 1) that distinguishes two families of item difficulty prediction approaches. Expert-driven approaches are qualitative, resting on "an expert consensus that exploits their domain knowledge and experience" and on pre-defined features from educational taxonomies or experts' difficulty perceptions. Machine-driven approaches quantify difficulty using statistical or data-driven prediction models to enhance scalability and minimise human intervention. The review uses this taxonomy to organise the field and to motivate its focus on automatic, text-based, a priori prediction. (Samah AlKhuzaey et al. (2023))

Claims

• Zero-shot GPT-4o difficulty ratings, rescaled and entered in grade-specific regressions, beat a grade-level dummy regressor and correlate strongly with true Rasch difficulties overall, for K-5 math and reading items [+W]
• LLMs systematically underestimate fraction items whose difficulty is driven by misconceptions (the \"Easy Trap\") [+W]

Related Research Methods

• Machine-learning prediction of accessible assessment-item quality

Key Sources

• Samah AlKhuzaey, Floriana Grasso, Terry R. Payne, Valentina Tamma. (2023). Text-based Question Difficulty Prediction: A Systematic Review of Automatic Approaches. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00362-1

Page metadata

Field	Value
type	research-method
id	taxonomy-of-item-difficulty-prediction-approaches-expert-driven-versus-machine-driven
title	Taxonomy of item difficulty prediction approaches: expert-driven versus machine-driven
description	The review introduces a classification (its Fig.
status	review
generated	by process:settle-candidates on 2026-10-28
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.