  Recently, there has been a growing surge of interest in enabling machine
learning systems to generalize well to Out-of-Distribution (OOD) data. Most
efforts are devoted to advancing optimization objectives that regularize models
to capture the underlying invariance; however, there often are compromises in
the optimization process of these OOD objectives: i) Many OOD objectives have
to be relaxed as penalty terms of Empirical Risk Minimization (ERM) for the
ease of optimization, while the relaxed forms can weaken the robustness of the
original objective; ii) The penalty terms also require careful tuning of the
penalty weights due to the intrinsic conflicts between ERM and OOD objectives.
Consequently, these compromises could easily lead to suboptimal performance of
either the ERM or OOD objective. To address these issues, we introduce a
multi-objective optimization (MOO) perspective to understand the OOD
optimization process, and propose a new optimization scheme called PAreto
Invariant Risk Minimization (PAIR). PAIR improves the robustness of OOD
objectives by cooperatively optimizing with other OOD objectives, thereby
bridging the gaps caused by the relaxations. Then PAIR approaches a Pareto
optimal solution that trades off the ERM and OOD objectives properly. Extensive
experiments on challenging benchmarks, WILDS, show that PAIR alleviates the
compromises and yields top OOD performances.
