  This paper describes noisy speech recognition for an augmented reality
headset that helps verbal communication within real multiparty conversational
environments. A major approach that has actively been studied in simulated
environments is to sequentially perform speech enhancement and automatic speech
recognition (ASR) based on deep neural networks (DNNs) trained in a supervised
manner. In our task, however, such a pretrained system fails to work due to the
mismatch between the training and test conditions and the head movements of the
user. To enhance only the utterances of a target speaker, we use beamforming
based on a DNN-based speech mask estimator that can adaptively extract the
speech components corresponding to a head-relative particular direction. We
propose a semi-supervised adaptation method that jointly updates the mask
estimator and the ASR model at run-time using clean speech signals with
ground-truth transcriptions and noisy speech signals with highly-confident
estimated transcriptions. Comparative experiments using the state-of-the-art
distant speech recognition system show that the proposed method significantly
improves the ASR performance.
