  Large language models (LLMs) hold tremendous potential for addressing
numerous real-world challenges, yet they typically demand significant
computational resources and memory. Deploying LLMs onto a resource-limited
hardware device with restricted memory capacity presents considerable
challenges. Distributed computing emerges as a prevalent strategy to mitigate
single-node memory constraints and expedite LLM inference performance. To
reduce the hardware limitation burden, we proposed an efficient distributed
inference optimization solution for LLMs on CPUs. We conduct experiments with
the proposed solution on 5th Gen Intel Xeon Scalable Processors, and the result
shows the time per output token for the LLM with 72B parameter is 140 ms/token,
much faster than the average human reading speed about 200ms per token.
