  Diabetic retinopathy (DR) is a complication of diabetes and usually takes
decades to reach sight-threatening levels. Accurate and robust detection of DR
severity is critical for the timely management and treatment of diabetes.
However, most current DR grading methods suffer from insufficient robustness to
data variability (\textit{e.g.} colour fundus images), posing a significant
difficulty for accurate and robust grading. In this work, we propose a novel DR
grading framework CLIP-DR based on three observations: 1) Recent pre-trained
visual language models, such as CLIP, showcase a notable capacity for
generalisation across various downstream tasks, serving as effective baseline
models. 2) The grading of image-text pairs for DR often adheres to a
discernible natural sequence, yet most existing DR grading methods have
primarily overlooked this aspect. 3) A long-tailed distribution among DR
severity levels complicates the grading process. This work proposes a novel
ranking-aware prompting strategy to help the CLIP model exploit the ordinal
information. Specifically, we sequentially design learnable prompts between
neighbouring text-image pairs in two different ranking directions.
Additionally, we introduce a Similarity Matrix Smooth module into the structure
of CLIP to balance the class distribution. Finally, we perform extensive
comparisons with several state-of-the-art methods on the GDRBench benchmark,
demonstrating our CLIP-DR's robustness and superior performance. The
implementation code is available
\footnote{\url{https://github.com/Qinkaiyu/CLIP-DR}
