Bloom-driven cognitive-leap evaluation framework with CogShift, category consistency, and knowledge coverage metrics¶
Research Method · All research methods
Evidence · 5 claims (4 for, 1 mixed) · 8 studies (4 causal, 3 design, 1 associational),q2· 0 of 8 report an effect size · 2 claims rest on one study
Description¶
A Bloom's-Taxonomy-driven framework for evaluating LLM-generated educational questions that pivots "from a knowledge-point perspective to a cognitive-leap paradigm". It combines usability filtering (unique, readable, answerable via majority voting across GPT evaluators), category consistency (CatCons), knowledge coverage (KnowCov), and a Cognitive Shift Score (CogShift) that weights mismatches by the ordinal distance between original and generated Bloom levels, classifying shifts as leap, regression, or drift. The framework was applied to 20,700 questions across six LLMs and three domains.
Accounts¶
- Bloom-driven cognitive-leap evaluation framework with CogShift, category consistency, and knowledge coverage metrics: A Bloom's-Taxonomy-driven framework for evaluating LLM-generated educational questions that pivots "from a knowledge-point perspective to a cognitive-leap paradigm". It combines usability filtering (unique, readable, answerable via majority voting across GPT evaluators), category consistency (CatCons), knowledge coverage (KnowCov), and a Cognitive Shift Score (CogShift) that weights mismatches by the ordinal distance between original and generated Bloom levels, classifying shifts as leap, regression, or drift. The framework was applied to 20,700 questions across six LLMs and three domains. (From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational)
Claims¶
- Cognitive leap is the most prevalent deviation pattern of LLM-generated questions, and fine-grained prompting amplifies upward shifts while suppressing cognitive regression [+M]
- Fine-grained prompting improves Bloom-level category consistency and knowledge coverage of generated questions in the computer science domain [+M]
- The Bloom-driven framework generalizes to social-science questions, with automated rubrics agreeing strongly with experts on usability but only modestly on Bloom level [+M]
- LLMs exhibit an inherent abstraction bias, consistently generating questions at higher-than-intended cognitive levels regardless of prompting strategy [~W]
- Higher knowledge-identification accuracy is associated with improved knowledge coverage in LLM-generated questions under Chain-of-Thought prompting [+W]
Related Research Methods¶
-
Key Sources¶
- Xiaolong Wang, Zhe Zhao, Song Lai, Chaoli Zhang, Zijie Geng, Yu Tong, Ye Wei, and Qingsong Wen. 2026. From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD '26). https://doi.org/10.1145/3770854.3785686
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
bloom-driven-cognitive-leap-evaluation-framework-with-cogshift-category-consistency-and-kn |
title |
Bloom-driven cognitive-leap evaluation framework with CogShift, category consistency, and knowledge coverage metrics |
description |
A Bloom's-Taxonomy-driven framework for evaluating LLM-generated educational questions that pivots "from a knowledge-point perspective to a cognitive-leap paradigm". |
status |
review |
generated |
by process:settle-candidates on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.