  A semantic equivalence assessment is defined as a task that assesses semantic
equivalence in a sentence pair by binary judgment (i.e., paraphrase
identification) or grading (i.e., semantic textual similarity measurement). It
constitutes a set of tasks crucial for research on natural language
understanding. Recently, BERT realized a breakthrough in sentence
representation learning (Devlin et al., 2019), which is broadly transferable to
various NLP tasks. While BERT's performance improves by increasing its model
size, the required computational power is an obstacle preventing practical
applications from adopting the technology. Herein, we propose to inject phrasal
paraphrase relations into BERT in order to generate suitable representations
for semantic equivalence assessment instead of increasing the model size.
Experiments on standard natural language understanding tasks confirm that our
method effectively improves a smaller BERT model while maintaining the model
size. The generated model exhibits superior performance compared to a larger
BERT model on semantic equivalence assessment tasks. Furthermore, it achieves
larger performance gains on tasks with limited training datasets for
fine-tuning, which is a property desirable for transfer learning.
