  Label embedding plays an important role in many real-world applications. To
enhance the label relatedness captured by the embeddings, multiple contexts can
be adopted. However, these contexts are heterogeneous and often partially
observed in practical tasks, imposing significant challenges to capture the
overall relatedness among labels. In this paper, we propose a general Partial
Heterogeneous Context Label Embedding (PHCLE) framework to address these
challenges. Categorizing heterogeneous contexts into two groups, relational
context and descriptive context, we design tailor-made matrix factorization
formula to effectively exploit the label relatedness in each context. With a
shared embedding principle across heterogeneous contexts, the label relatedness
is selectively aligned in a shared space. Due to our elegant formulation, PHCLE
overcomes the partial context problem and can nicely incorporate more contexts,
which both cannot be tackled with existing multi-context label embedding
methods. An effective alternative optimization algorithm is further derived to
solve the sparse matrix factorization problem. Experimental results demonstrate
that the label embeddings obtained with PHCLE achieve superb performance in
image classification task and exhibit good interpretability in the downstream
label similarity analysis and image understanding task.
