  Much progress has been made in the deep neural network (DNN) based diagnosis
of mass lesions breast ultrasound (BUS) images. However, the non-mass lesion is
less investigated because of the limited data. Based on the insight that mass
data is sufficient and shares the same knowledge structure with non-mass data
of identifying the malignancy of a lesion based on the ultrasound image, we
propose a novel transfer learning framework to enhance the generalizability of
the DNN model for non-mass BUS with the help of mass BUS. Specifically, we
train a shared DNN with combined non-mass and mass data. With the prior of
different marginal distributions in input and output space, we employ two
domain alignment strategies in the proposed transfer learning framework with
the insight of capturing domain-specific distribution to address the issue of
domain shift. Moreover, we propose a cross-domain semantic-preserve data
generation module called CrossMix to recover the missing distribution between
non-mass and mass data that is not presented in training data. Experimental
results on an in-house dataset demonstrate that the DNN model trained with
combined data by our framework achieves a 10% improvement in AUC on the
malignancy prediction task of non-mass BUS compared to training directly on
non-mass data.
