  Book covers are usually the very first impression to its readers and they
often convey important information about the content of the book. Book genre
classification based on its cover would be utterly beneficial to many modern
retrieval systems, considering that the complete digitization of books is an
extremely expensive task. At the same time, it is also an extremely challenging
task due to the following reasons: First, there exists a wide variety of book
genres, many of which are not concretely defined. Second, book covers, as
graphic designs, vary in many different ways such as colors, styles, textual
information, etc, even for books of the same genre. Third, book cover designs
may vary due to many external factors such as country, culture, target reader
populations, etc. With the growing competitiveness in the book industry, the
book cover designers and typographers push the cover designs to its limit in
the hope of attracting sales. The cover-based book classification systems
become a particularly exciting research topic in recent years. In this paper,
we propose a multi-modal deep learning framework to solve this problem. The
contribution of this paper is four-fold. First, our method adds an extra
modality by extracting texts automatically from the book covers. Second,
image-based and text-based, state-of-the-art models are evaluated thoroughly
for the task of book cover classification. Third, we develop an efficient and
salable multi-modal framework based on the images and texts shown on the covers
only. Fourth, a thorough analysis of the experimental results is given and
future works to improve the performance is suggested. The results show that the
multi-modal framework significantly outperforms the current state-of-the-art
image-based models. However, more efforts and resources are needed for this
classification task in order to reach a satisfactory level.
