  The Prostate Imaging Reporting and Data System (PI-RADS) is pivotal in the
diagnosis of clinically significant prostate cancer through MRI imaging.
Current deep learning-based PI-RADS scoring methods often lack the
incorporation of common PI-RADS clinical guideline~(PICG) utilized by
radiologists, potentially compromising scoring accuracy. This paper introduces
a novel approach that adapts a multi-modal large language model (MLLM) to
incorporate PICG into PI-RADS scoring model without additional annotations and
network parameters. We present a designed two-stage fine-tuning process aiming
at adapting a MLLM originally trained on natural images to the MRI images while
effectively integrating the PICG. Specifically, in the first stage, we develop
a domain adapter layer tailored for processing 3D MRI inputs and instruct the
MLLM to differentiate MRI sequences. In the second stage, we translate PICG for
guiding instructions from the model to generate PICG-guided image features.
Through such a feature distillation step, we align the scoring network's
features with the PICG-guided image features, which enables the model to
effectively incorporate the PICG information. We develop our model on a public
dataset and evaluate it on an in-house dataset. Experimental results
demonstrate that our approach effectively improves the performance of current
scoring networks. Code is available at: https://github.com/med-air/PICG2scoring
