  Recently, the use of pre-trained model to build neural network based on
transfer learning methodology is increasingly popular. These pre-trained models
present the benefit of using less computing resources to train model with
smaller amount of training data. The rise of state-of-the-art models such as
BERT, XLNet and GPT boost accuracy and benefit as a base model for transfer
leanring. However, these models are still too complex and consume many
computing resource to train for transfer learning with low GPU memory. We will
compare the performance and cost between lighter transfer learning model and
purposely built neural network for NLP application of text classification and
NER model.
