  Being expensive and time-consuming to collect massive COVID-19 image samples
to train deep classification models, transfer learning is a promising approach
by transferring knowledge from the abundant typical pneumonia datasets for
COVID-19 image classification. However, negative transfer may deteriorate the
performance due to the feature distribution divergence between two datasets and
task semantic difference in diagnosing pneumonia and COVID-19 that rely on
different characteristics. It is even more challenging when the target dataset
has no labels available, i.e., unsupervised task transfer learning. In this
paper, we propose a novel Task Adaptation Network (TAN) to solve this
unsupervised task transfer problem. In addition to learning transferable
features via domain-adversarial training, we propose a novel task semantic
adaptor that uses the learning-to-learn strategy to adapt the task semantics.
Experiments on three public COVID-19 datasets demonstrate that our proposed
method achieves superior performance. Especially on COVID-DA dataset, TAN
significantly increases the recall and F1 score by 5.0% and 7.8% compared to
recently strong baselines. Moreover, we show that TAN also achieves superior
performance on several public domain adaptation benchmarks.
