  Non-autoregressive mechanisms can significantly decrease inference time for
speech transformers, especially when the single step variant is applied.
Previous work on CTC alignment-based single step non-autoregressive transformer
(CASS-NAT) has shown a large real time factor (RTF) improvement over
autoregressive transformers (AT). In this work, we propose several methods to
improve the accuracy of the end-to-end CASS-NAT, followed by performance
analyses. First, convolution augmented self-attention blocks are applied to
both the encoder and decoder modules. Second, we propose to expand the trigger
mask (acoustic boundary) for each token to increase the robustness of CTC
alignments. In addition, iterated loss functions are used to enhance the
gradient update of low-layer parameters. Without using an external language
model, the WERs of the improved CASS-NAT, when using the three methods, are
3.1%/7.2% on Librispeech test clean/other sets and the CER is 5.4% on the
Aishell1 test set, achieving a 7%~21% relative WER/CER improvement. For the
analyses, we plot attention weight distributions in the decoders to visualize
the relationships between token-level acoustic embeddings. When the acoustic
embeddings are visualized, we find that they have a similar behavior to word
embeddings, which explains why the improved CASS-NAT performs similarly to AT.
