  Previous works indicate that the glyph of Chinese characters contains rich
semantic information and has the potential to enhance the representation of
Chinese characters. The typical method to utilize the glyph features is by
incorporating them into the character embedding space. Inspired by previous
methods, we innovatively propose a Chinese pre-trained representation model
named as GlyphCRM, which abandons the ID-based character embedding method yet
solely based on sequential character images. We render each character into a
binary grayscale image and design two-channel position feature maps for it.
Formally, we first design a two-layer residual convolutional neural network,
namely HanGlyph to generate the initial glyph representation of Chinese
characters, and subsequently adopt multiple bidirectional encoder Transformer
blocks as the superstructure to capture the context-sensitive information.
Meanwhile, we feed the glyph features extracted from each layer of the HanGlyph
module into the underlying Transformer blocks by skip-connection method to
fully exploit the glyph features of Chinese characters. As the HanGlyph module
can obtain a sufficient glyph representation of any Chinese character, the
long-standing out-of-vocabulary problem could be effectively solved. Extensive
experimental results indicate that GlyphCRM substantially outperforms the
previous BERT-based state-of-the-art model on 9 fine-tuning tasks, and it has
strong transferability and generalization on specialized fields and
low-resource tasks. We hope this work could spark further research beyond the
realms of well-established representation of Chinese texts.
