  In recent years, large language models have demonstrated remarkable
performance across various natural language processing (NLP) tasks. However,
deploying these models for real-world applications often requires efficient
inference solutions to handle the computational demands. In this paper, we
explore the utilization of CPUs for accelerating the inference of large
language models. Specifically, we introduce a parallelized approach to enhance
throughput by 1) Exploiting the parallel processing capabilities of modern CPU
architectures, 2) Batching the inference request. Our evaluation shows the
accelerated inference engine gives an 18-22x improvement in the generated token
per sec. The improvement is more with longer sequence and larger models. In
addition to this, we can also run multiple workers in the same machine with
NUMA node isolation to further improvement in tokens/s. Table 2, we have
received 4x additional improvement with 4 workers. This would also make Gen-AI
based products and companies environment friendly, our estimates shows that CPU
usage for Inference could reduce the power consumption of LLMs by 48.9% while
providing production ready throughput and latency.
