  Service manual documents are crucial to the engineering company as they
provide guidelines and knowledge to service engineers. However, it has become
inconvenient and inefficient for service engineers to retrieve specific
knowledge from documents due to the complexity of resources. In this research,
we propose an automated knowledge mining and document classification system
with novel multi-model transfer learning approaches. Particularly, the
classification performance of the system has been improved with three effective
techniques: fine-tuning, pruning, and multi-model method. The fine-tuning
technique optimizes a pre-trained BERT model by adding a feed-forward neural
network layer and the pruning technique is used to retrain the BERT model with
new data. The multi-model method initializes and trains multiple BERT models to
overcome the randomness of data ordering during the fine-tuning process. In the
first iteration of the training process, multiple BERT models are being trained
simultaneously. The best model is then selected for the next phase of the
training process with another two iterations and the training processes for
other BERT models will be terminated. The performance of the proposed system
has been evaluated by comparing with two robust baseline methods, BERT and
BERT-CNN. Experimental results on a widely used Corpus of Linguistic
Acceptability (CoLA) dataset have shown that the proposed techniques perform
better than these baseline methods in terms of accuracy and MCC score.
