  Modern speech recognition systems exhibits rapid performance degradation
under domain shift. This issue is especially prevalent in data-scarce settings,
such as low-resource languages, where diversity of training data is limited. In
this work we propose M2DS2, a simple and sample-efficient finetuning strategy
for large pretrained speech models, based on mixed source and target domain
self-supervision. We find that including source domain self-supervision
stabilizes training and avoids mode collapse of the latent representations. For
evaluation, we collect HParl, a $120$ hour speech corpus for Greek, consisting
of plenary sessions in the Greek Parliament. We merge HParl with two popular
Greek corpora to create GREC-MD, a test-bed for multi-domain evaluation of
Greek ASR systems. In our experiments we find that, while other Unsupervised
Domain Adaptation baselines fail in this resource-constrained environment,
M2DS2 yields significant improvements for cross-domain adaptation, even when a
only a few hours of in-domain audio are available. When we relax the problem in
a weakly supervised setting, we find that independent adaptation for audio
using M2DS2 and language using simple LM augmentation techniques is
particularly effective, yielding word error rates comparable to the fully
supervised baselines.
