  We present a new benchmark dataset called PARADE for paraphrase
identification that requires specialized domain knowledge. PARADE contains
paraphrases that overlap very little at the lexical and syntactic level but are
semantically equivalent based on computer science domain knowledge, as well as
non-paraphrases that overlap greatly at the lexical and syntactic level but are
not semantically equivalent based on this domain knowledge. Experiments show
that both state-of-the-art neural models and non-expert human annotators have
poor performance on PARADE. For example, BERT after fine-tuning achieves an F1
score of 0.709, which is much lower than its performance on other paraphrase
identification datasets. PARADE can serve as a resource for researchers
interested in testing models that incorporate domain knowledge. We make our
data and code freely available.
