  Transformer-based end-to-end speech recognition models have received
considerable attention in recent years due to their high training speed and
ability to model a long-range global context. Position embedding in the
transformer architecture is indispensable because it provides supervision for
dependency modeling between elements at different positions in the input
sequence. To make use of the time order of the input sequence, many works
inject some information about the relative or absolute position of the element
into the input sequence. In this work, we investigate various position
embedding methods in the convolution-augmented transformer (conformer) and
adopt a novel implementation named rotary position embedding (RoPE). RoPE
encodes absolute positional information into the input sequence by a rotation
matrix, and then naturally incorporates explicit relative position information
into a self-attention module. To evaluate the effectiveness of the RoPE method,
we conducted experiments on AISHELL-1 and LibriSpeech corpora. Results show
that the conformer enhanced with RoPE achieves superior performance in the
speech recognition task. Specifically, our model achieves a relative word error
rate reduction of 8.70% and 7.27% over the conformer on test-clean and
test-other sets of the LibriSpeech corpus respectively.
