  Self-supervised speech representation learning enables the extraction of
meaningful features from raw waveforms. These features can then be efficiently
used across multiple downstream tasks. However, two significant issues arise
when considering the deployment of such methods ``in-the-wild": (i) Their large
size, which can be prohibitive for edge applications; and (ii) their robustness
to detrimental factors, such as noise and/or reverberation, that can heavily
degrade the performance of such systems. In this work, we propose
RobustDistiller, a novel knowledge distillation mechanism that tackles both
problems jointly. Simultaneously to the distillation recipe, we apply a
multi-task learning objective to encourage the network to learn noise-invariant
representations by denoising the input. The proposed mechanism is evaluated on
twelve different downstream tasks. It outperforms several benchmarks regardless
of noise type, or noise and reverberation levels. Experimental results show
that the new Student model with 23M parameters can achieve results comparable
to the Teacher model with 95M parameters. Lastly, we show that the proposed
recipe can be applied to other distillation methodologies, such as the recent
DPWavLM. For reproducibility, code and model checkpoints will be made available
at \mbox{\url{https://github.com/Hguimaraes/robustdistiller}}.
