  When deploying machine learning systems to the wild, it is highly desirable
for them to effectively leverage prior knowledge to the unfamiliar domain while
also firing alarms to anomalous inputs. In order to address these requirements,
Universal Domain Adaptation (UniDA) has emerged as a novel research area in
computer vision, focusing on achieving both adaptation ability and robustness
(i.e., the ability to detect out-of-distribution samples). While UniDA has led
significant progress in computer vision, its application on language input
still needs to be explored despite its feasibility. In this paper, we propose a
comprehensive benchmark for natural language that offers thorough viewpoints of
the model's generalizability and robustness. Our benchmark encompasses multiple
datasets with varying difficulty levels and characteristics, including temporal
shifts and diverse domains. On top of our testbed, we validate existing UniDA
methods from computer vision and state-of-the-art domain adaptation techniques
from NLP literature, yielding valuable findings: We observe that UniDA methods
originally designed for image input can be effectively transferred to the
natural language domain while also underscoring the effect of adaptation
difficulty in determining the model's performance.
