  Transfer learning improves the performance of deep learning models by
initializing them with parameters pre-trained on larger datasets. Intuitively,
transfer learning is more effective when pre-training is on the in-domain
datasets. A recent study by NASA has demonstrated that the microstructure
segmentation with encoder-decoder algorithms benefits more from CNN encoders
pre-trained on microscopy images than from those pre-trained on natural images.
However, CNN models only capture the local spatial relations in images. In
recent years, attention networks such as Transformers are increasingly used in
image analysis to capture the long-range relations between pixels. In this
study, we compare the segmentation performance of Transformer and CNN models
pre-trained on microscopy images with those pre-trained on natural images. Our
result partially confirms the NASA study that the segmentation performance of
out-of-distribution images (taken under different imaging and sample
conditions) is significantly improved when pre-training on microscopy images.
However, the performance gain for one-shot and few-shot learning is more modest
with Transformers. We also find that for image segmentation, the combination of
pre-trained Transformers and CNN encoders are consistently better than
pre-trained CNN encoders alone. Our dataset (of about 50,000 images) combines
the public portion of the NASA dataset with additional images we collected.
Even with much less training data, our pre-trained models have significantly
better performance for image segmentation. This result suggests that
Transformers and CNN complement each other and when pre-trained on microscopy
images, they are more beneficial to the downstream tasks.
