  Traditional Bag-of-visual Words (BoWs) model is commonly generated with many
steps including local feature extraction, codebook generation, and feature
quantization, etc. Those steps are relatively independent with each other and
are hard to be jointly optimized. Moreover, the dependency on hand-crafted
local feature makes BoWs model not effective in conveying high-level semantics.
These issues largely hinder the performance of BoWs model in large-scale image
applications. To conquer these issues, we propose an End-to-End BoWs
(E$^2$BoWs) model based on Deep Convolutional Neural Network (DCNN). Our model
takes an image as input, then identifies and separates the semantic objects in
it, and finally outputs the visual words with high semantic discriminative
power. Specifically, our model firstly generates Semantic Feature Maps (SFMs)
corresponding to different object categories through convolutional layers, then
introduces Bag-of-Words Layers (BoWL) to generate visual words for each
individual feature map. We also introduce a novel learning algorithm to
reinforce the sparsity of the generated E$^2$BoWs model, which further ensures
the time and memory efficiency. We evaluate the proposed E$^2$BoWs model on
several image search datasets including CIFAR-10, CIFAR-100, MIRFLICKR-25K and
NUS-WIDE. Experimental results show that our method achieves promising accuracy
and efficiency compared with recent deep learning based retrieval works.
