  Citrinet is an end-to-end convolutional Connectionist Temporal Classification
(CTC) based automatic speech recognition (ASR) model. To capture local and
global contextual information, 1D time-channel separable convolutions combined
with sub-word encoding and squeeze-and-excitation (SE) are used in Citrinet,
making the whole architecture to be as deep as including 23 blocks with 235
convolution layers and 46 linear layers. This pure convolutional and deep
architecture makes Critrinet relatively slow at convergence. In this paper, we
propose to introduce multi-head attentions together with feed-forward networks
in the convolution module in Citrinet blocks while keeping the SE module and
residual module unchanged. For speeding up, we remove 8 convolution layers in
each attention-enhanced Citrinet block and reduce 23 blocks to 13. Experiments
on the Japanese CSJ-500h and Magic-1600h dataset show that the
attention-enhanced Citrinet with less layers and blocks and converges faster
with lower character error rates than (1) Citrinet with 80\% training time and
(2) Conformer with 40\% training time and 29.8\% model size.
