  Wav2vec2.0 is a popular self-supervised pre-training framework for learning
speech representations in the context of automatic speech recognition (ASR). It
was shown that wav2vec2.0 has a good robustness against the domain shift, while
the noise robustness is still unclear. In this work, we therefore first analyze
the noise robustness of wav2vec2.0 via experiments. We observe that wav2vec2.0
pre-trained on noisy data can obtain good representations and thus improve the
ASR performance on the noisy test set, which however brings a performance
degradation on the clean test set. To avoid this issue, in this work we propose
an enhanced wav2vec2.0 model. Specifically, the noisy speech and the
corresponding clean version are fed into the same feature encoder, where the
clean speech provides training targets for the model. Experimental results
reveal that the proposed method can not only improve the ASR performance on the
noisy test set which surpasses the original wav2vec2.0, but also ensure a tiny
performance decrease on the clean test set. In addition, the effectiveness of
the proposed method is demonstrated under different types of noise conditions.
