  While significant improvements have been made in recent years in terms of
end-to-end automatic speech recognition (ASR) performance, such improvements
were obtained through the use of very large neural networks, unfit for embedded
use on edge devices. That being said, in this paper, we work on simplifying and
compressing Transformer-based encoder-decoder architectures for the end-to-end
ASR task. We empirically introduce a more compact Speech-Transformer by
investigating the impact of discarding particular modules on the performance of
the model. Moreover, we evaluate reducing the numerical precision of our
network's weights and activations while maintaining the performance of the
full-precision model. Our experiments show that we can reduce the number of
parameters of the full-precision model and then further compress the model 4x
by fully quantizing to 8-bit fixed point precision.
