  The automatic classification is a process of automatically assigning text
documents to predefined categories. An accurate automatic patent classifier is
crucial to patent inventors and patent examiners in terms of intellectual
property protection, patent management, and patent information retrieval. We
present BERT-CNN, a hierarchical patent classifier based on pre-trained
language model by training the national patent application documents collected
from the State Information Center, China. The experimental results show that
BERT-CNN achieves 84.3% accuracy, which is far better than the two compared
baseline methods, Convolutional Neural Networks and Recurrent Neural Networks.
We didn't apply our model to the third and fourth hierarchical level of the
International Patent Classification - "subclass" and "group".The visualization
of the Attention Mechanism shows that BERT-CNN obtains new state-of-the-art
results in representing vocabularies and semantics. This article demonstrates
the practicality and effectiveness of BERT-CNN in the field of automatic patent
classification.
