  Product descriptions in e-commerce platforms contain detailed and valuable
information about retailers assortment. In particular, coding promotions within
digital leaflets are of great interest in e-commerce as they capture the
attention of consumers by showing regular promotions for different products.
However, this information is embedded into images, making it difficult to
extract and process for downstream tasks. In this paper, we present an
end-to-end approach that classifies promotions within digital leaflets into
their corresponding product categories using both visual and textual
information. Our approach can be divided into three key components: 1) region
detection, 2) text recognition and 3) text classification. In many cases, a
single promotion refers to multiple product categories, so we introduce a
multi-label objective in the classification head. We demonstrate the
effectiveness of our approach for two separated tasks: 1) image-based detection
of the descriptions for each individual promotion and 2) multi-label
classification of the product categories using the text from the product
descriptions. We train and evaluate our models using a private dataset composed
of images from digital leaflets obtained by Nielsen. Results show that we
consistently outperform the proposed baseline by a large margin in all the
experiments.
