  This paper describes the University of Sydney& JD's joint submission of the
IWSLT 2021 low resource speech translation task. We participated in the
Swahili-English direction and got the best scareBLEU (25.3) score among all the
participants. Our constrained system is based on a pipeline framework, i.e. ASR
and NMT. We trained our models with the officially provided ASR and MT
datasets. The ASR system is based on the open-sourced tool Kaldi and this work
mainly explores how to make the most of the NMT models. To reduce the
punctuation errors generated by the ASR model, we employ our previous work
SlotRefine to train a punctuation correction model. To achieve better
translation performance, we explored the most recent effective strategies,
including back translation, knowledge distillation, multi-feature reranking and
transductive finetuning. For model structure, we tried auto-regressive and
non-autoregressive models, respectively. In addition, we proposed two novel
pre-train approaches, i.e. \textit{de-noising training} and
\textit{bidirectional training} to fully exploit the data. Extensive
experiments show that adding the above techniques consistently improves the
BLEU scores, and the final submission system outperforms the baseline
(Transformer ensemble model trained with the original parallel data) by
approximately 10.8 BLEU score, achieving the SOTA performance.
