  Term frequency-inverse document frequency, or TF-IDF for short, and its many
variants form a class of term weighting functions the members of which are
widely used in text analysis applications. While TF-IDF was originally proposed
as a heuristic, theoretical justifications grounded in information theory,
probability, and the divergence from randomness paradigm have been advanced. In
this work, we present an empirical study showing that TF-IDF corresponds very
nearly with the hypergeometric test of statistical significance on selected
real-data document retrieval, summarization, and classification tasks. These
findings suggest that a fundamental mathematical connection between TF-IDF and
the negative logarithm of the hypergeometric test P-value (i.e., a
hypergeometric distribution tail probability) remains to be elucidated. We
advance the empirical analyses herein as a first step toward explaining the
long-standing effectiveness of TF-IDF from a statistical significance testing
lens. It is our aspiration that these results will open the door to the
systematic evaluation of significance testing derived term weighting functions
in text analysis applications.
