  Domain Adaptation (DA) has received widespread attention from deep learning
researchers in recent years because of its potential to improve test accuracy
with out-of-distribution labeled data. Most state-of-the-art DA algorithms
require an extensive amount of hyperparameter tuning and are computationally
intensive due to the large batch sizes required. In this work, we propose a
fast and simple DA method consisting of three stages: (1) domain alignment by
covariance matching, (2) pseudo-labeling, and (3) ensembling. We call this
method $\textbf{PACE}$, for $\textbf{P}$seudo-labels, $\textbf{A}$lignment of
$\textbf{C}$ovariances, and $\textbf{E}$nsembles. PACE is trained on top of
fixed features extracted from an ensemble of modern pretrained backbones. PACE
exceeds previous state-of-the-art by $\textbf{5 - 10 \%}$ on most benchmark
adaptation tasks without training a neural network. PACE reduces training time
and hyperparameter tuning time by $82\%$ and $97\%$, respectively, when
compared to state-of-the-art DA methods. Code is released here:
https://github.com/Chris210634/PACE-Domain-Adaptation
