Five-part question taxonomy with multi-endpoint metrics for evaluating course-grounded AI tutors¶
Research Method · All research methods
Evidence · 2 claims (2 for) · 2 studies (1 causal, 1 design),q2· 0 of 2 report an effect size · 2 claims rest on one study
Description¶
The article extends a prior 30-question generic/specific set into a five-part taxonomy of classroom question types: "(a) single fact lookup, (b) multi hop concept linking, (c) thematic explanation and synthesis, (d) contradictionandambiguityhandling(Houetal.2024),and (e) curriculum update questions probing sensitivity to revisedmaterials." Questions were human-generated with reference answers and word limits and balanced across course topics, yielding 59 retained questions (45 topic-page, 14 cross-page). The taxonomy is paired with a multi-endpoint metric suite — RAGAS core metrics plus claim-citation alignment, blinded LLM-judge synthesis quality, robustness under knowledge shift, and cost — scored separately to avoid winner-take-all conclusions.
Accounts¶
- Five-part question taxonomy with multi-endpoint metrics for evaluating course-grounded AI tutors: The article extends a prior 30-question generic/specific set into a five-part taxonomy of classroom question types: "(a) single fact lookup, (b) multi hop concept linking, (c) thematic explanation and synthesis, (d) contradictionandambiguityhandling(Houetal.2024),and (e) curriculum update questions probing sensitivity to revisedmaterials." Questions were human-generated with reference answers and word limits and balanced across course topics, yielding 59 retained questions (45 topic-page, 14 cross-page). The taxonomy is paired with a multi-endpoint metric suite — RAGAS core metrics plus claim-citation alignment, blinded LLM-judge synthesis quality, robustness under knowledge shift, and cost — scored separately to avoid winner-take-all conclusions. (Brian Wright (2026))
Claims¶
- The wiki's advantage over vector RAG is roughly three times larger on cross-page questions than on topic-page questions, in both score and groundedness [+M]
- LLM graders show model-specific precision/recall trade-offs, so a single metric is insufficient to judge grading quality [+W]
Related Research Methods¶
-
Key Sources¶
- Brian Wright. (2026). Potential for Enhanced Learning in Machine Learning Classes by Using Wiki LLM Indexing. https://arxiv.org/abs/2609.25303
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
five-part-question-taxonomy-with-multi-endpoint-metrics-for-evaluating-course-grounded-ai |
title |
Five-part question taxonomy with multi-endpoint metrics for evaluating course-grounded AI tutors |
description |
The article extends a prior 30-question generic/specific set into a five-part taxonomy of classroom question types: "(a) single fact lookup, (b) multi hop concept linking, (c) thematic explanation and synthesis, (d) contradictionandambiguityhandling(Houetal.2024),and (e) curriculum update questions |
status |
review |
generated |
by process:settle-candidates on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.