  The rapidly evolving literature of COVID-19 related articles makes it
challenging for NLP models to be effectively trained for information retrieval
and extraction with the corresponding labeled data that follows the current
distribution of the pandemic. On the other hand, due to the uncertainty of the
situation, human experts' supervision would always be required to double check
the decision making of these models highlighting the importance of
interpretability. In the light of these challenges, this study proposes an
interpretable self-supervised multi-task learning model to jointly and
effectively tackle the tasks of information retrieval (IR) and extraction (IE)
during the current emergency health crisis situation. Our results show that our
model effectively leverage the multi-task and self-supervised learning to
improve generalization, data efficiency and robustness to the ongoing dataset
shift problem. Our model outperforms baselines in IE and IR tasks, respectively
by micro-f score of 0.08 (LCA-F score of 0.05), and MAP of 0.05 on average. In
IE the zero- and few-shot learning performances are on average 0.32 and 0.19
micro-f score higher than those of the baselines.
