  In this paper, we propose a hybrid text normalization system using multi-head
self-attention. The system combines the advantages of a rule-based model and a
neural model for text preprocessing tasks. Previous studies in Mandarin text
normalization usually use a set of hand-written rules, which are hard to
improve on general cases. The idea of our proposed system is motivated by the
neural models from recent studies and has a better performance on our internal
news corpus. This paper also includes different attempts to deal with
imbalanced pattern distribution of the dataset. Overall, the performance of the
system is improved by over 1.5% on sentence-level and it has a potential to
improve further.
