  Many Natural Language Processing and Computational Linguistics applications
involves the generation of new texts based on some existing texts, such as
summarization, text simplification and machine translation. However, there has
been a serious problem haunting these applications for decades, that is, how to
automatically and accurately assess quality of these applications. In this
paper, we will present some preliminary results on one especially useful and
challenging problem in NLP system evaluation: how to pinpoint content
differences of two text passages (especially for large pas-sages such as
articles and books). Our idea is intuitive and very different from existing
approaches. We treat one text passage as a small knowledge base, and ask it a
large number of questions to exhaustively identify all content points in it. By
comparing the correctly answered questions from two text passages, we will be
able to compare their content precisely. The experiment using 2007 DUC
summarization corpus clearly shows promising results.
