  Few-shot learning has been studied to adapt models to tasks with very few
samples. It holds profound significance, particularly in clinical tasks, due to
the high annotation cost of medical images. Several works have explored
few-shot learning on medical images, yet they still require a large number of
medical images for pre-training models to gain domain-specific priors. Vision
foundation models recently have achieved remarkable success in natural images.
Hence, adapting rapidly advancing vision foundation models from natural images
to few-shot clinical tasks holds great promise. MedFMC has recently organized a
challenge to shed more light on this topic at NeurIPS 2023. In this work, we
present our challenge solution. We observe that a simple variant of fine-tuning
with partial freezing shows remarkable performance. Empirical evidence
demonstrates that this approach could outperform various common fine-tuning
methods under limited sample sizes. Additionally, we explore enhanced
utilization of semantic supervision to boost performance. We propose a novel
approach that contextualizes labels via large language models (LLMs). Our
findings reveal that the context generated by LLMs significantly enhances the
discrimination of semantic embeddings for similar categories, resulting in a
notable performance improvement of 3%-5% in 1-shot settings compared to
commonly employed one-hot labels and other semantic supervision methods. Our
solution secures the 1st place in the MedFMC challenge.
