  In this paper, we present a novel method that reduces model inference latency
during distributed deployment of Large Language Models (LLMs). Our contribution
is an optimized inference deployment scheme that address the current
limitations of state-of-the-art quantization kernels when used in conjunction
with Tensor Parallel (TP). Our method preserves data locality in GPU memory
access patterns and exploits a priori knowledge of TP to reduce global
communication. We demonstrate an up to 1.81x speedup over existing methods for
Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
