  Recent work on discriminative segmental models has shown that they can
achieve competitive speech recognition performance, using features based on
deep neural frame classifiers. However, segmental models can be more
challenging to train than standard frame-based approaches. While some segmental
models have been successfully trained end to end, there is a lack of
understanding of their training under different settings and with different
losses.
  We investigate a model class based on recent successful approaches,
consisting of a linear model that combines segmental features based on an LSTM
frame classifier. Similarly to hybrid HMM-neural network models, segmental
models of this class can be trained in two stages (frame classifier training
followed by linear segmental model weight training), end to end (joint training
of both frame classifier and linear weights), or with end-to-end fine-tuning
after two-stage training.
  We study segmental models trained end to end with hinge loss, log loss,
latent hinge loss, and marginal log loss. We consider several losses for the
case where training alignments are available as well as where they are not.
  We find that in general, marginal log loss provides the most consistent
strong performance without requiring ground-truth alignments. We also find that
training with dropout is very important in obtaining good performance with
end-to-end training. Finally, the best results are typically obtained by a
combination of two-stage training and fine-tuning.
