Ablation study shows metadata-enriched embeddings produce the largest accuracy drop, while attention masking, KV caching and dynamic batching primarily support inference efficiency¶

Claim · All claims
Evidence · 2 studies · 1 causal r2, 1 design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-22 (automated, 2 of 2 entries) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Removing metadata-enriched embeddings drops overall accuracy from 69.8% to 65.3%, the largest decline among tested ablations. → Shen X 2026 q2 i? Disabling curriculum-specific attention masking raises energy consumption from 2.0 to 2.4 mWh/query with only minor accuracy effect. → Shen X 2026 (2)

Evidence¶

Shen X 2026¶

Shen X, Feng L, Hua S, Liu D, Xie Z and Liu B (2026) Towards sustainable AI knowledge-base assistants in computer science education: on-premise deployment and optimization with open educational resources. Front. Psychol. 17:1843444. https://doi.org/10.3389/fpsyg.2026.1843444

q2 · i? · design · r2

Single-factor ablation study (Section 5.4, Table 6) using Qwen-7B (Ours) as base configuration, each ablation disabling one component on the full n = 300 query set. The article reports the drop "from 69.8 to 65.3%". No standardized effect size is printed.

"Removing metadata-enriched embeddings produces the largest accuracy drop, from 69.8 to 65.3%, indicating that structured pedagogical metadata materially improves retrieval quality and downstream response accuracy"

Shen X 2026 (2)¶

Shen X, Feng L, Hua S, Liu D, Xie Z and Liu B (2026) Towards sustainable AI knowledge-base assistants in computer science education: on-premise deployment and optimization with open educational resources. Front. Psychol. 17:1843444. https://doi.org/10.3389/fpsyg.2026.1843444

q2 · i? · causal · r2

Same ablation study (Table 6): disabling attention masking, KV caching, or dynamic batching raised energy or latency with negligible accuracy change. The article states masking's "primary contribution is computational efficiency". No standardized effect size is printed.

"curriculum-specific attention masking has only a minor effect on accuracy but increases energy consumption from 2.0 to 2.4 mWh/query when removed, indicating that its primary contribution is computational efficiency rather than answer quality"

Discussion¶

Related Claims¶

• Static Knowledge Grounding and Dynamic Personal Memory are complementary: removing both yields the largest quality degradation — related
• Removing the cognitive load module causes the largest ablation performance drop, while state fusion has a smaller effect — related
• Pipeline components play complementary roles: hierarchical generation has the largest effect, grounding improves specificity and factual accuracy, and cluster-informed TOC planning improves structure and audience fit — related
• Ablation study: removing any component lowers evaluation AUC, and removing all additional features yields the lowest public and private AUCs — related
• Knowledge and behavior profile components provide complementary signals: removing either hurts metrics tied to the other — related
• Federated aggregation is the primary driver of performance gains, contributing an average 11.99% improvement — related
• Ablating expert-informed structural components lowered LLM-judge scores on all dimensions, with largest drops in challenge quality, intent alignment, and domain grounding — related
• Each pedagogical verification component contributes to a distinct quality dimension: layout verification most strongly — related
• DeepSeek-MoE (Ours) achieves the strongest overall balance of accuracy among tested configurations, reaching 78.6% overall and 81.2% multi-hop reasoning accuracy on a 300-query OER evaluation set — related
• Ablation baselines were configured to isolate the contribution of cross-modal attention, PPO clipping and Kalman filtering, but component-level results are not reported in the available text — related
• Ablations show the trustworthiness module contributes most to performance, and the explanation generator cuts human review time by 41% — related