  This paper investigates how to improve the runtime speed of personalized
speech enhancement (PSE) networks while maintaining the model quality. Our
approach includes two aspects: architecture and knowledge distillation (KD). We
propose an end-to-end enhancement (E3Net) model architecture, which is
$3\times$ faster than a baseline STFT-based model. Besides, we use KD
techniques to develop compressed student models without significantly degrading
quality. In addition, we investigate using noisy data without reference clean
signals for training the student models, where we combine KD with multi-task
learning (MTL) using automatic speech recognition (ASR) loss. Our results show
that E3Net provides better speech and transcription quality with a lower target
speaker over-suppression (TSOS) rate than the baseline model. Furthermore, we
show that the KD methods can yield student models that are $2-4\times$ faster
than the teacher and provides reasonable quality. Combining KD and MTL improves
the ASR and TSOS metrics without degrading the speech quality.
