  Self-attention-based transformer models have achieved tremendous success in
the domain of natural language processing. Despite their efficacy, accelerating
the transformer is challenging due to its quadratic computational complexity
and large activation sizes. Existing transformer accelerators attempt to prune
its tokens to reduce memory access, albeit with high compute overheads.
Moreover, previous works directly operate on large matrices involved in the
attention operation, which limits hardware utilization. In order to address
these challenges, this work proposes a novel dynamic inference scheme,
DynaTran, which prunes activations at runtime with low overhead, substantially
reducing the number of ineffectual operations. This improves the throughput of
transformer inference. We further propose tiling the matrices in transformer
operations along with diverse dataflows to improve data reuse, thus enabling
higher energy efficiency. To effectively implement these methods, we propose
AccelTran, a novel accelerator architecture for transformers. Extensive
experiments with different models and benchmarks demonstrate that DynaTran
achieves higher accuracy than the state-of-the-art top-k hardware-aware pruning
strategy while attaining up to 1.2$\times$ higher sparsity. One of our proposed
accelerators, AccelTran-Edge, achieves 330K$\times$ higher throughput with
93K$\times$ lower energy requirement when compared to a Raspberry Pi device. On
the other hand, AccelTran-Server achieves 5.73$\times$ higher throughput and
3.69$\times$ lower energy consumption compared to the state-of-the-art
transformer co-processor, Energon. The simulation source code is available at
https://github.com/jha-lab/acceltran.
