  Pre-trained Language Models (PLMs) have achieved remarkable performance gains
across numerous downstream tasks in natural language understanding. Various
Chinese PLMs have been successively proposed for learning better Chinese
language representation. However, most current models use Chinese characters as
inputs and are not able to encode semantic information contained in Chinese
words. While recent pre-trained models incorporate both words and characters
simultaneously, they usually suffer from deficient semantic interactions and
fail to capture the semantic relation between words and characters. To address
the above issues, we propose a simple yet effective PLM CLOWER, which adopts
the Contrastive Learning Over Word and charactER representations. In
particular, CLOWER implicitly encodes the coarse-grained information (i.e.,
words) into the fine-grained representations (i.e., characters) through
contrastive learning on multi-grained information. CLOWER is of great value in
realistic scenarios since it can be easily incorporated into any existing
fine-grained based PLMs without modifying the production pipelines.Extensive
experiments conducted on a range of downstream tasks demonstrate the superior
performance of CLOWER over several state-of-the-art baselines.
