  Code-switching (CS) is common in daily conversations where more than one
language is used within a sentence. The difficulties of CS speech recognition
lie in alternating languages and the lack of transcribed data. Therefore, this
paper uses the recently successful self-supervised learning (SSL) methods to
leverage many unlabeled speech data without CS. We show that hidden
representations of SSL models offer frame-level language identity even if the
models are trained with English speech only. Jointly training CTC and language
identification modules with self-supervised speech representations improves CS
speech recognition performance. Furthermore, using multilingual speech data for
pre-training obtains the best CS speech recognition.
