  Despite the emergence of principled methods for domain adaptation under label
shift, their sensitivity to shifts in class conditional distributions is
precariously under explored. Meanwhile, popular deep domain adaptation
heuristics tend to falter when faced with label proportions shifts. While
several papers modify these heuristics in attempts to handle label proportions
shifts, inconsistencies in evaluation standards, datasets, and baselines make
it difficult to gauge the current best practices. In this paper, we introduce
RLSbench, a large-scale benchmark for relaxed label shift, consisting of $>$500
distribution shift pairs spanning vision, tabular, and language modalities,
with varying label proportions. Unlike existing benchmarks, which primarily
focus on shifts in class-conditional $p(x|y)$, our benchmark also focuses on
label marginal shifts. First, we assess 13 popular domain adaptation methods,
demonstrating more widespread failures under label proportion shifts than were
previously known. Next, we develop an effective two-step meta-algorithm that is
compatible with most domain adaptation heuristics: (i) pseudo-balance the data
at each epoch; and (ii) adjust the final classifier with target label
distribution estimate. The meta-algorithm improves existing domain adaptation
heuristics under large label proportion shifts, often by 2--10\% accuracy
points, while conferring minimal effect ($<$0.5\%) when label proportions do
not shift. We hope that these findings and the availability of RLSbench will
encourage researchers to rigorously evaluate proposed methods in relaxed label
shift settings. Code is publicly available at
https://github.com/acmi-lab/RLSbench.
