  We study the problem of automatically building hypernym taxonomies from
textual and visual data. Previous works in taxonomy induction generally ignore
the increasingly prominent visual data, which encode important perceptual
semantics. Instead, we propose a probabilistic model for taxonomy induction by
jointly leveraging text and images. To avoid hand-crafted feature engineering,
we design end-to-end features based on distributed representations of images
and words. The model is discriminatively trained given a small set of existing
ontologies and is capable of building full taxonomies from scratch for a
collection of unseen conceptual label items with associated images. We evaluate
our model and features on the WordNet hierarchies, where our system outperforms
previous approaches by a large gap.
