  Existing skin attributes detection methods usually initialize with a
pre-trained Imagenet network and then fine-tune on a medical target task.
However, we argue that such approaches are suboptimal because medical datasets
are largely different from ImageNet and often contain limited training samples.
In this work, we propose \emph{Task Agnostic Transfer Learning (TATL)}, a novel
framework motivated by dermatologists' behaviors in the skincare context. TATL
learns an attribute-agnostic segmenter that detects lesion skin regions and
then transfers this knowledge to a set of attribute-specific classifiers to
detect each particular attribute. Since TATL's attribute-agnostic segmenter
only detects skin attribute regions, it enjoys ample data from all attributes,
allows transferring knowledge among features, and compensates for the lack of
training data from rare attributes. We conduct extensive experiments to
evaluate the proposed TATL transfer learning mechanism with various neural
network architectures on two popular skin attributes detection benchmarks. The
empirical results show that TATL not only works well with multiple
architectures but also can achieve state-of-the-art performances while enjoying
minimal model and computational complexities. We also provide theoretical
insights and explanations for why our transfer learning framework performs well
in practice.
