  The development of methods to deal with the informative contents of the text
units in the matching process is a major challenge in automatic summary
evaluation systems that use fixed n-gram matching. The limitation causes
inaccurate matching between units in a peer and reference summaries. The
present study introduces a new Keyphrase based Summary Evaluator KpEval for
evaluating automatic summaries. The KpEval relies on the keyphrases since they
convey the most important concepts of a text. In the evaluation process, the
keyphrases are used in their lemma form as the matching text unit. The system
was applied to evaluate different summaries of Arabic multi-document data set
presented at TAC2011. The results showed that the new evaluation technique
correlates well with the known evaluation systems: Rouge1, Rouge2, RougeSU4,
and AutoSummENG MeMoG. KpEval has the strongest correlation with AutoSummENG
MeMoG, Pearson and spearman correlation coefficient measures are 0.8840, 0.9667
respectively.
