  A popular approach to semantic image understanding is to manually tag images
with keywords and then learn a mapping from vi- sual features to keywords.
Manually tagging images is a subjective pro- cess and the same or very similar
visual contents are often tagged with different keywords. Furthermore, not all
tags have the same descriptive power for visual contents and large vocabulary
available from natural language could result in a very diverse set of keywords.
In this paper, we propose an unsupervised visual theme discovery framework as a
better (more compact, efficient and effective) alternative to semantic
represen- tation of visual contents. We first show that tag based annotation
lacks consistency and compactness for describing visually similar contents. We
then learn the visual similarity between tags based on the visual features of
the images containing the tags. At the same time, we use a natural language
processing technique (word embedding) to measure the seman- tic similarity
between tags. Finally, we cluster tags into visual themes based on their visual
similarity and semantic similarity measures using a spectral clustering
algorithm. We conduct user studies to evaluate the effectiveness and
rationality of the visual themes discovered by our unsu- pervised algorithm and
obtains promising result. We then design three common computer vision tasks,
example based image search, keyword based image search and image labelling to
explore potential applica- tion of our visual themes discovery framework. In
experiments, visual themes significantly outperforms tags on semantic image
understand- ing and achieve state-of-art performance in all three tasks. This
again demonstrate the effectiveness and versatility of proposed framework.
