  The importance of an efficient and scalable document similarity detection
system is undeniable nowadays. Search engines need batch text similarity
measures to detect duplicated and near-duplicated web pages in their indexes in
order to prevent indexing a web page multiple times. Furthermore, in the
scoring phase, search engines need similarity measures to detect duplicated
contents on web pages so as to increase the quality of their results. In this
paper, a new approach to batch text similarity detection is proposed by
combining some ideas from dimensionality reduction techniques and information
gain theory. The new approach is focused on search engines need to detect
duplicated and near-duplicated web pages. The new approach is evaluated on the
NEWS20 dataset and the results show that the new approach is faster than the
cosine text similarity algorithm in terms of speed and performance. On top of
that, It is faster and more accurate than the other rival method, Simhash
similarity algorithm.
