  Recognizing arbitrary objects in the wild has been a challenging problem due
to the limitations of existing classification models and datasets. In this
paper, we propose a new task that aims at parsing scenes with a large and open
vocabulary, and several evaluation metrics are explored for this problem. Our
proposed approach to this problem is a joint image pixel and word concept
embeddings framework, where word concepts are connected by semantic relations.
We validate the open vocabulary prediction ability of our framework on ADE20K
dataset which covers a wide variety of scenes and objects. We further explore
the trained joint embedding space to show its interpretability.
