  Models pre-trained on large-scale regular text corpora often do not work well
for user-generated data where the language styles differ significantly from the
mainstream text. Here we present Context-Aware Rule Injection (CARI), an
innovative method for formality style transfer (FST). CARI injects multiple
rules into an end-to-end BERT-based encoder and decoder model. It learns to
select optimal rules based on context. The intrinsic evaluation showed that
CARI achieved the new highest performance on the FST benchmark dataset. Our
extrinsic evaluation showed that CARI can greatly improve the regular
pre-trained models' performance on several tweet sentiment analysis tasks.
