  Many self-supervised learning (SSL) methods have been successful in learning
semantically meaningful visual representations by solving pretext tasks.
However, prior work in SSL focuses on tasks like object recognition or
detection, which aim to learn object shapes and assume that the features should
be invariant to concepts like colors and textures. Thus, these SSL methods
perform poorly on downstream tasks where these concepts provide critical
information. In this paper, we present an SSL framework that enables us to
learn color and texture-aware features without requiring any labels during
training. Our approach consists of three self-supervised tasks designed to
capture different concepts that are neglected in prior work that we can select
from depending on the needs of our downstream tasks. Our tasks include learning
to predict color histograms and discriminate shapeless local patches and
textures from each instance. We evaluate our approach on fashion compatibility
using Polyvore Outfits and In-Shop Clothing Retrieval using Deepfashion,
improving upon prior SSL methods by 9.5-16%, and even outperforming some
supervised approaches on Polyvore Outfits despite using no labels. We also show
that our approach can be used for transfer learning, demonstrating that we can
train on one dataset while achieving high performance on a different dataset.
