  We study semantic collisions: texts that are semantically unrelated but
judged as similar by NLP models. We develop gradient-based approaches for
generating semantic collisions and demonstrate that state-of-the-art models for
many tasks which rely on analyzing the meaning and similarity of texts--
including paraphrase identification, document retrieval, response suggestion,
and extractive summarization-- are vulnerable to semantic collisions. For
example, given a target query, inserting a crafted collision into an irrelevant
document can shift its retrieval rank from 1000 to top 3. We show how to
generate semantic collisions that evade perplexity-based filtering and discuss
other potential mitigations. Our code is available at
https://github.com/csong27/collision-bert.
