  Current contrastive learning frameworks focus on leveraging a single
supervisory signal to learn representations, which limits the efficacy on
unseen data and downstream tasks. In this paper, we present a hierarchical
multi-label representation learning framework that can leverage all available
labels and preserve the hierarchical relationship between classes. We introduce
novel hierarchy preserving losses, which jointly apply a hierarchical penalty
to the contrastive loss, and enforce the hierarchy constraint. The loss
function is data driven and automatically adapts to arbitrary multi-label
structures. Experiments on several datasets show that our
relationship-preserving embedding performs well on a variety of tasks and
outperform the baseline supervised and self-supervised approaches. Code is
available at https://github.com/salesforce/hierarchicalContrastiveLearning.
