  Representational transfer from publicly available models is a promising
technique for improving medical image classification, especially in long-tailed
datasets with rare diseases. However, existing methods often overlook the
frequency-dependent behavior of these models, thereby limiting their
effectiveness in transferring representations and generalizations to rare
diseases. In this paper, we propose FoPro-KD, a novel framework that leverages
the power of frequency patterns learned from frozen pre-trained models to
enhance their transferability and compression, presenting a few unique
insights: 1) We demonstrate that leveraging representations from publicly
available pre-trained models can substantially improve performance,
specifically for rare classes, even when utilizing representations from a
smaller pre-trained model. 2) We observe that pre-trained models exhibit
frequency preferences, which we explore using our proposed Fourier Prompt
Generator (FPG), allowing us to manipulate specific frequencies in the input
image, enhancing the discriminative representational transfer. 3) By amplifying
or diminishing these frequencies in the input image, we enable Effective
Knowledge Distillation (EKD). EKD facilitates the transfer of knowledge from
pre-trained models to smaller models. Through extensive experiments in
long-tailed gastrointestinal image recognition and skin lesion classification,
where rare diseases are prevalent, our FoPro-KD framework outperforms existing
methods, enabling more accessible medical models for rare disease
classification. Code is available at https://github.com/xmed-lab/FoPro-KD.
