Efficient Near-Duplicate Document Detection using FPGAs
Detecting duplicate and near-duplicate documents is critical in applications like Web crawling since it helps save document processing resources. Simhash is a state-of-art method to assign a bit-string fingerprint to a document, such that similar documents have similar fingerprints. Finding the near-duplicates in a large collection of documents consists of two stages: compute the simhash fingerprint of each document, and find pairs of similar fingerprints by computing their Hamming distance. Previous paper has focused on optimizing the second stage, i.e., avoiding the quadratic number of comparisons to compute the all to all Hamming distance.