  Although fusion of information from multiple views of mammograms plays an
important role to increase accuracy of breast cancer detection, developing
multi-view mammograms-based computer-aided diagnosis (CAD) schemes still faces
challenges and no such CAD schemes have been used in clinical practice. To
overcome the challenges, we investigate a new approach based on Contrastive
Language-Image Pre-training (CLIP), which has sparked interest across various
medical imaging tasks. By solving the challenges in (1) effectively adapting
the single-view CLIP for multi-view feature fusion and (2) efficiently
fine-tuning this parameter-dense model with limited samples and computational
resources, we introduce Mammo-CLIP, the first multi-modal framework to process
multi-view mammograms and corresponding simple texts. Mammo-CLIP uses an early
feature fusion strategy to learn multi-view relationships in four mammograms
acquired from the CC and MLO views of the left and right breasts. To enhance
learning efficiency, plug-and-play adapters are added into CLIP image and text
encoders for fine-tuning parameters and limiting updates to about 1% of the
parameters. For framework evaluation, we assembled two datasets
retrospectively. The first dataset, comprising 470 malignant and 479 benign
cases, was used for few-shot fine-tuning and internal evaluation of the
proposed Mammo-CLIP via 5-fold cross-validation. The second dataset, including
60 malignant and 294 benign cases, was used to test generalizability of
Mammo-CLIP. Study results show that Mammo-CLIP outperforms the state-of-art
cross-view transformer in AUC (0.841 vs. 0.817, 0.837 vs. 0.807) on both
datasets. It also surpasses previous two CLIP-based methods by 20.3% and 14.3%.
This study highlights the potential of applying the finetuned vision-language
models for developing next-generation, image-text-based CAD schemes of breast
cancer.
