State-of-the-art embedding-based keyphrase models degrade on lecture slides while TF-IDF generalizes better to them¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? TF-IDF achieves higher F1-scores on lecture slides than on general documents, indicating robustness across domains. → Erwin Daniel López Zapata 2025 q2 i? AttentionRank shows a pronounced drop in performance on lecture slides compared with its results on general NLP datasets. → Erwin Daniel López Zapata 2025 (2)

Evidence¶

Erwin Daniel López Zapata 2025¶

Erwin Daniel López Zapata, Cheng Tang, Valdemar Švábenský, Fumiya Okubo, Atsushi Shimada. (2025). LECTOR: Summarizing E-book Reading Content for Personalized Student Support. Int J Artif Intell Educ 35. https://doi.org/10.1007/s40593-025-00478-6

q2 · i? · design · r2

Comparison of F1-scores for keyphrase extraction on benchmark general-document datasets versus the lecture-slides dataset (Table 6); the article attributes TF-IDF's result to "TF-IDF's robustness and generalizability across domains".

"TF-IDF, despite being a simple statistical model, achieves higher performance on lecture slides than on general documents (Table 6)."

Erwin Daniel López Zapata 2025 (2)¶

Erwin Daniel López Zapata, Cheng Tang, Valdemar Švábenský, Fumiya Okubo, Atsushi Shimada. (2025). LECTOR: Summarizing E-book Reading Content for Personalized Student Support. Int J Artif Intell Educ 35. https://doi.org/10.1007/s40593-025-00478-6

q2 · i? · design · r2

Same benchmark comparison (Table 6): AttentionRank's F1 at @5 falls from 24.02 on general documents to 17.25 on lecture slides; the article concludes that "recent advances in unsupervised keyphrase extraction may not adapt well to educational contexts".

"This suggests that recent advances in unsupervised keyphrase extraction may not adapt well to educational contexts, especially for extracting key concepts from lecture slides."

Discussion¶

Related Claims¶

• LECTOR achieves the highest keyphrase extraction performance among compared models at @15, Best and Mean evaluations on lecture slides — reports the opposite
• Students rated LECTOR-extracted keyphrases as more relevant than TF-IDF-extracted keyphrases in a human evaluation — related
• Under the fixed protocol, hosted LLMs produce the largest gain on Spanish thematic classification, with thematic macro F1 rising from 0.46 (TF-IDF) to roughly 0.82-0.86 for the two hosted LLMs — related
• Feature-based ML Bloom classifiers trained on IID data degrade sharply on out-of-distribution AI-assisted questions — related