  We consider the problem of training speech recognition systems without using
any labeled data, under the assumption that the learner can only access to the
input utterances and a phoneme language model estimated from a non-overlapping
corpus. We propose a fully unsupervised learning algorithm that alternates
between solving two sub-problems: (i) learn a phoneme classifier for a given
set of phoneme segmentation boundaries, and (ii) refining the phoneme
boundaries based on a given classifier. To solve the first sub-problem, we
introduce a novel unsupervised cost function named Segmental Empirical Output
Distribution Matching, which generalizes the work in (Liu et al., 2017) to
segmental structures. For the second sub-problem, we develop an approximate MAP
approach to refining the boundaries obtained from Wang et al. (2017).
Experimental results on TIMIT dataset demonstrate the success of this fully
unsupervised phoneme recognition system, which achieves a phone error rate
(PER) of 41.6%. Although it is still far away from the state-of-the-art
supervised systems, we show that with oracle boundaries and matching language
model, the PER could be improved to 32.5%.This performance approaches the
supervised system of the same model architecture, demonstrating the great
potential of the proposed method.
