  Segmental models are sequence prediction models in which scores of hypotheses
are based on entire variable-length segments of frames. We consider segmental
models for whole-word ("acoustic-to-word") speech recognition, with the feature
vectors defined using vector embeddings of segments. Such models are
computationally challenging as the number of paths is proportional to the
vocabulary size, which can be orders of magnitude larger than when using
subword units like phones. We describe an efficient approach for end-to-end
whole-word segmental models, with forward-backward and Viterbi decoding
performed on a GPU and a simple segment scoring function that reduces space
complexity. In addition, we investigate the use of pre-training via jointly
trained acoustic word embeddings (AWEs) and acoustically grounded word
embeddings (AGWEs) of written word labels. We find that word error rate can be
reduced by a large margin by pre-training the acoustic segment representation
with AWEs, and additional (smaller) gains can be obtained by pre-training the
word prediction layer with AGWEs. Our final models improve over prior A2W
models.
