  We present IndoBERTweet, the first large-scale pretrained model for
Indonesian Twitter that is trained by extending a monolingually-trained
Indonesian BERT model with additive domain-specific vocabulary. We focus in
particular on efficient model adaptation under vocabulary mismatch, and
benchmark different ways of initializing the BERT embedding layer for new word
types. We find that initializing with the average BERT subword embedding makes
pretraining five times faster, and is more effective than proposed methods for
vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based
datasets.
