  Transformers have revolutionized deep learning and generative modeling to
enable unprecedented advancements in natural language processing tasks and
beyond. However, designing hardware accelerators for executing transformer
models is challenging due to the wide variety of computing kernels involved in
the transformer architecture. Existing accelerators are either inadequate to
accelerate end-to-end transformer models or suffer notable thermal limitations.
In this paper, we propose the design of a three-dimensional heterogeneous
architecture referred to as HeTraX specifically optimized to accelerate
end-to-end transformer models. HeTraX employs hardware resources aligned with
the computational kernels of transformers and optimizes both performance and
energy. Experimental results show that HeTraX outperforms existing
state-of-the-art by up to 5.6x in speedup and improves EDP by 14.5x while
ensuring thermally feasibility.
