  Quantization techniques are pivotal in reducing the memory and computational
demands of deep neural network inference. Existing solutions, such as
ZeroQuant, offer dynamic quantization for models like BERT and GPT but overlook
crucial memory-bounded operators and the complexities of per-token
quantization. Addressing these gaps, we present a novel, fully
hardware-enhanced robust optimized post-training W8A8 quantization framework,
ZeroQuant-HERO. This framework uniquely integrates both memory bandwidth and
compute-intensive operators, aiming for optimal hardware performance.
Additionally, it offers flexibility by allowing specific INT8 modules to switch
to FP16/BF16 mode, enhancing accuracy.
