  Early detection of dysplasia of the cervix is critical for cervical cancer
treatment. However, automatic cervical dysplasia diagnosis via visual
inspection, which is more appropriate in low-resource settings, remains a
challenging problem. Though promising results have been obtained by recent deep
learning models, their performance is significantly hindered by the limited
scale of the available cervix datasets. Distinct from previous methods that
learn from a single dataset, we propose to leverage cross-domain cervical
images that were collected in different but related clinical studies to improve
the model's performance on the targeted cervix dataset. To robustly learn the
transferable information across datasets, we propose a novel prototype-based
knowledge filtering method to estimate the transferability of cross-domain
samples. We further optimize the shared feature space by aligning the
cross-domain image representations simultaneously on domain level with early
alignment and class level with supervised contrastive learning, which endows
model training and knowledge transfer with stronger robustness. The empirical
results on three real-world benchmark cervical image datasets show that our
proposed method outperforms the state-of-the-art cervical dysplasia visual
inspection by an absolute improvement of 4.7% in top-1 accuracy, 7.0% in
precision, 1.4% in recall, 4.6% in F1 score, and 0.05 in ROC-AUC.
