  Discovering automatically the semantic structure of tagged visual data (e.g.
web videos and images) is important for visual data analysis and
interpretation, enabling the machine intelligence for effectively processing
the fast-growing amount of multi-media data. However, this is non-trivial due
to the need for jointly learning underlying correlations between heterogeneous
visual and tag data. The task is made more challenging by inherently sparse and
incomplete tags. In this work, we develop a method for modelling the inherent
visual data concept structures based on a novel Hierarchical-Multi-Label Random
Forest model capable of correlating structured visual and tag information so as
to more accurately interpret the visual semantics, e.g. disclosing meaningful
visual groups with similar high-level concepts, and recovering missing tags for
individual visual data samples. Specifically, our model exploits hierarchically
structured tags of different semantic abstractness and multiple tag statistical
correlations in addition to modelling visual and tag interactions. As a result,
our model is able to discover more accurate semantic correlation between
textual tags and visual features, and finally providing favourable visual
semantics interpretation even with highly sparse and incomplete tags. We
demonstrate the advantages of our proposed approach in two fundamental
applications, visual data clustering and missing tag completion, on
benchmarking video (i.e. TRECVID MED 2011) and image (i.e. NUS-WIDE) datasets.
