  The success in designing Code-Switching (CS) ASR often depends on the
availability of the transcribed CS resources. Such dependency harms the
development of ASR in low-resourced languages such as Bengali and Hindi. In
this paper, we exploit the transfer learning approach to design End-to-End
(E2E) CS ASR systems for the two low-resourced language pairs using different
monolingual speech data and a small set of noisy CS data. We trained the
CS-ASR, following two steps: (i) building a robust bilingual ASR system using a
convolution-augmented transformer (Conformer) based acoustic model and n-gram
language model, and (ii) fine-tuned the entire E2E ASR with limited noisy CS
data. We tested our method on MUCS 2021 challenge and achieved 3rd place in the
CS track. We then tested the proposed method using noisy CS data released for
Hindi-English and Bengali-English pairs in Multilingual and Code-Switching ASR
Challenges for Low Resource Indian Languages (MUCS 2021) and achieved 3rd place
in the CS track. Unlike, the leading two systems that benefited from crawling
YouTube and learning transliteration pairs, our proposed transfer learning
approach focused on using only the limited CS data with no data-cleaning or
data re-segmentation. Our approach achieved 14.1% relative gain in word error
rate (WER) in Hindi-English and 27.1% in Bengali-English. We provide detailed
guidelines on the steps to finetune the self-attention based model for limited
data for ASR. Moreover, we release the code and recipe used in this paper.
