  This study compares the effectiveness and robustness of multi-class
categorization of Amazon product data using transfer learning on pre-trained
contextualized language models. Specifically, we fine-tuned BERT and XLNet, two
bidirectional models that have achieved state-of-the-art performance on many
natural language tasks and benchmarks, including text classification. While
existing classification studies and benchmarks focus on binary targets, with
the exception of ordinal ranking tasks, here we examine the robustness of such
models as the number of classes grows from 1 to 20. Our experiments demonstrate
an approximately linear decrease in performance metrics (i.e., precision,
recall, $F_1$ score, and accuracy) with the number of class labels. BERT
consistently outperforms XLNet using identical hyperparameters on the entire
range of class label quantities for categorizing products based on their
textual descriptions. BERT is also more affordable than XLNet in terms of the
computational cost (i.e., time and memory) required for training. In all cases
studied, the performance degradation rates were estimated to be 1% per
additional class label.
