  Concept-based models naturally lend themselves to the development of
inherently interpretable skin lesion diagnosis, as medical experts make
decisions based on a set of visual patterns of the lesion. Nevertheless, the
development of these models depends on the existence of concept-annotated
datasets, whose availability is scarce due to the specialized knowledge and
expertise required in the annotation process. In this work, we show that
vision-language models can be used to alleviate the dependence on a large
number of concept-annotated samples. In particular, we propose an embedding
learning strategy to adapt CLIP to the downstream task of skin lesion
classification using concept-based descriptions as textual embeddings. Our
experiments reveal that vision-language models not only attain better accuracy
when using concepts as textual embeddings, but also require a smaller number of
concept-annotated samples to attain comparable performance to approaches
specifically devised for automatic concept generation.
