  One of the important factors that make a search engine fast and accurate is a
concise and duplicate free index. In order to remove duplicate and
near-duplicate documents from the index, a search engine needs a swift and
reliable duplicate and near-duplicate text document detection system.
Traditional approaches to this problem, such as brute force comparisons or
simple hash-based algorithms are not suitable as they are not scalable and are
not capable of detecting near-duplicate documents effectively. In this paper, a
new signature-based approach to text similarity detection is introduced which
is fast, scalable, reliable and needs less storage space. The proposed method
is examined on popular text document data-sets such as CiteseerX, Enron, Gold
Set of Near-duplicate News Articles and etc. The results are promising and
comparable with the best cutting-edge algorithms, considering the accuracy and
performance. The proposed method is based on the idea of using reference texts
to generate signatures for text documents. The novelty of this paper is the use
of genetic algorithms to generate better reference texts.
