  The model's ability to understand synonymous expression is crucial in many
kinds of downstream tasks. It will make the model to better understand the
similarity between context, and more robust to the synonym substitution attack.
However, many Pretrained Language Model (PLM) lack synonym knowledge due to
limitation of small-scale synsets and PLM's pretraining objectives. In this
paper, we propose a framework called Sem4SAP to mine synsets from Open
Knowledge Graph (Open-KG) and using the mined synsets to do synonym-aware
pretraining for language models. We propose to coarsly filter the content in
Open-KG and use the frequency information to better help the clustering process
under low-resource unsupervised conditions. We expand the mined synsets by
migrating core semantics between synonymous expressions.We also propose two
novel and effective synonym-aware pre-training methods for injecting synonym
knowledge into PLMs.Extensive experiments demonstrate that Sem4SAP can
dramatically outperform the original PLMs and other baselines on ten different
tasks.
