  This report presents an empirical evaluation of four algorithms for
automatically extracting keywords and keyphrases from documents. The four
algorithms are compared using five different collections of documents. For each
document, we have a target set of keyphrases, which were generated by hand. The
target keyphrases were generated for human readers; they were not tailored for
any of the four keyphrase extraction algorithms. Each of the algorithms was
evaluated by the degree to which the algorithm's keyphrases matched the
manually generated keyphrases. The four algorithms were (1) the AutoSummarize
feature in Microsoft's Word 97, (2) an algorithm based on Eric Brill's
part-of-speech tagger, (3) the Summarize feature in Verity's Search 97, and (4)
NRC's Extractor algorithm. For all five document collections, NRC's Extractor
yields the best match with the manually generated keyphrases.
