  Breast cancer is the most common invasive cancer in women. Besides the
primary B-mode ultrasound screening, sonographers have explored the inclusion
of Doppler, strain and shear-wave elasticity imaging to advance the diagnosis.
However, recognizing useful patterns in all types of images and weighing up the
significance of each modality can elude less-experienced clinicians. In this
paper, we explore, for the first time, an automatic way to combine the four
types of ultrasonography to discriminate between benign and malignant breast
nodules. A novel multimodal network is proposed, along with promising
learnability and simplicity to improve classification accuracy. The key is
using a weight-sharing strategy to encourage interactions between modalities
and adopting an additional cross-modalities objective to integrate global
information. In contrast to hardcoding the weights of each modality in the
model, we embed it in a Reinforcement Learning framework to learn this
weighting in an end-to-end manner. Thus the model is trained to seek the
optimal multimodal combination without handcrafted heuristics. The proposed
framework is evaluated on a dataset contains 1616 set of multimodal images.
Results showed that the model scored a high classification accuracy of 95.4%,
which indicates the efficiency of the proposed method.
