  Pretrained Language Models (PLM) have been greatly successful on a board
range of natural language processing (NLP) tasks. However, it has just started
being applied to the domain of recommendation systems. Traditional
recommendation algorithms failed to incorporate the rich textual information in
e-commerce datasets, which hinderss the performance of those models. We present
a thorough investigation on the effect of various strategy of incorporating
PLMs into traditional recommender algorithms on one of the e-commerce datasets,
and we compare the results with vanilla recommender baseline models. We show
that the application of PLMs and domain specific fine-tuning lead to an
increase on the predictive capability of combined models. These results
accentuate the importance of utilizing textual information in the context of
e-commerce, and provides insight on how to better apply PLMs alongside
traditional recommender system algorithms. The code used in this paper is
available on Github: https://github.com/NuofanXu/bert_retail_recommender.
