tf-idf term-weighting scheme from information retrieval¶

Element · All elements
Evidence · 3 claims (1 for, 2 against) · 3 studies (2 design, 1 associational), q2 · 1 of 3 report an effect size · 3 claims rest on one study

Description¶

tf-idf is a composite weighting scheme from information retrieval that combines a term's frequency within a document with its inverse document frequency across the collection, formalized as tf(t,d) times idf(t,D). The chapter states that "tf-idf selects for terms that occur many times in a small number of documents, penalizing terms that occur only a few times in a document, or that occur in many documents." It is used to identify effective keywords and provides the feature space for distributional models such as latent semantic indexing and Bayesian topic models, illustrating that context-sensitive dispersion measures are useful for retrieval systems and, by analogy, for understanding human lexical organization.

Design Implications¶

Context¶

Requirements¶

• An indexed document collection over which document-level dispersion statistics can be computed

Constraints¶

• The chapter notes the best keywords have high variance over documents and low document entropy, prioritizing bursty terms over uniformly distributed ones

Target Learners¶

• information seekers
• search system users

Target Learning Goals¶

• information retrieval
• keyword selection

Claims¶

• IDF reweighting of knowledge vectors improves retrieval over binary KC-incidence vectors, with the largest gains in mAP [+W]
• Under the fixed protocol, hosted LLMs produce the largest gain on Spanish thematic classification, with thematic macro F1 rising from 0.46 (TF-IDF) to roughly 0.82-0.86 for the two hosted LLMs [-W]
• Feature-based models (LLM-extracted features plus metadata in random forest and gradient boosting) outperform direct LLM estimation, dummy regressor, TF-IDF, and metadata-only benchmarks, with correlations up to r = .87 [-W]

Related Elements¶

Examples¶

-

Key Sources¶

• Jones, M. N., Dye, M., & Johns, B. T. (2017). Context as an Organizing Principle of the Lexicon. Psychology of Learning and Motivation, Volume 67. https://doi.org/10.1016/bs.plm.2017.03.008