  Recent advances on Out-of-Distribution (OoD) generalization reveal the
robustness of deep learning models against distribution shifts. However,
existing works focus on OoD algorithms, such as invariant risk minimization,
domain generalization, or stable learning, without considering the influence of
deep model architectures on OoD generalization, which may lead to sub-optimal
performance. Neural Architecture Search (NAS) methods search for architecture
based on its performance on the training data, which may result in poor
generalization for OoD tasks. In this work, we propose robust Neural
Architecture Search for OoD generalization (NAS-OoD), which optimizes the
architecture with respect to its performance on generated OoD data by gradient
descent. Specifically, a data generator is learned to synthesize OoD data by
maximizing losses computed by different neural architectures, while the goal
for architecture search is to find the optimal architecture parameters that
minimize the synthetic OoD data losses. The data generator and the neural
architecture are jointly optimized in an end-to-end manner, and the minimax
training process effectively discovers robust architectures that generalize
well for different distribution shifts. Extensive experimental results show
that NAS-OoD achieves superior performance on various OoD generalization
benchmarks with deep models having a much fewer number of parameters. In
addition, on a real industry dataset, the proposed NAS-OoD method reduces the
error rate by more than 70% compared with the state-of-the-art method,
demonstrating the proposed method's practicality for real applications.
