  Learning low-dimensional representation for large number of products present
in an e-commerce catalogue plays a vital role as they are helpful in tasks like
product ranking, product recommendation, finding similar products, modelling
user-behaviour etc. Recently, a lot of tasks in the NLP field are getting
tackled using the Transformer based models and these deep models are widely
applicable in the industries setting to solve various problems. With this
motivation, we apply transformer based model for learning contextual
representation of products in an e-commerce setting. In this work, we propose a
novel approach of pre-training transformer based model on a users generated
sessions dataset obtained from a large fashion e-commerce platform to obtain
latent product representation. Once pre-trained, we show that the low-dimension
representation of the products can be obtained given the product attributes
information as a textual sentence. We mainly pre-train BERT, RoBERTa, ALBERT
and XLNET variants of transformer model and show a quantitative analysis of the
products representation obtained from these models with respect to Next Product
Recommendation(NPR) and Content Ranking(CR) tasks. For both the tasks, we
collect an evaluation data from the fashion e-commerce platform and observe
that XLNET model outperform other variants with a MRR of 0.5 for NPR and NDCG
of 0.634 for CR. XLNET model also outperforms the Word2Vec based
non-transformer baseline on both the downstream tasks. To the best of our
knowledge, this is the first and novel work for pre-training transformer based
models using users generated sessions data containing products that are
represented with rich attributes information for adoption in e-commerce
setting. These models can be further fine-tuned in order to solve various
downstream tasks in e-commerce, thereby eliminating the need to train a model
from scratch.
