  This paper introduces a novel approach to learn visually grounded meaning
representations of words as low-dimensional node embeddings on an underlying
graph hierarchy. The lower level of the hierarchy models modality-specific word
representations through dedicated but communicating graphs, while the higher
level puts these representations together on a single graph to learn a
representation jointly from both modalities. The topology of each graph models
similarity relations among words, and is estimated jointly with the graph
embedding. The assumption underlying this model is that words sharing similar
meaning correspond to communities in an underlying similarity graph in a
low-dimensional space. We named this model Hierarchical Multi-Modal Similarity
Graph Embedding (HM-SGE). Experimental results validate the ability of HM-SGE
to simulate human similarity judgements and concept categorization,
outperforming the state of the art.
