  Novelty detection in text streams is a challenging task that emerges in quite
a few different scenarios, ranging from email thread filtering to RSS news feed
recommendation on a smartphone. An efficient novelty detection algorithm can
save the user a great deal of time and resources when browsing through relevant
yet usually previously-seen content. Most of the recent research on detection
of novel documents in text streams has been building upon either geometric
distances or distributional similarities, with the former typically performing
better but being much slower due to the need of comparing an incoming document
with all the previously-seen ones. In this paper, we propose a new approach to
novelty detection in text streams. We describe a resource-aware mechanism that
is able to handle massive text streams such as the ones present today thanks to
the burst of social media and the emergence of the Web as the main source of
information. We capitalize on the historical Inverse Document Frequency (IDF)
that was known for capturing well term specificity and we show that it can be
used successfully at the document level as a measure of document novelty. This
enables us to avoid similarity comparisons with previous documents in the text
stream, thus scaling better and leading to faster execution times. Moreover, as
the collection of documents evolves over time, we use a temporal variant of IDF
not only to maintain an efficient representation of what has already been seen
but also to decay the document frequencies as the time goes by. We evaluate the
performance of the proposed approach on a real-world news articles dataset
created for this task. The results show that the proposed method outperforms
all of the baselines while managing to operate efficiently in terms of time
complexity and memory usage, which are of great importance in a mobile setting
scenario.
