  On-device large language models (LLMs) are catalyzing novel mobile
applications such as UI task automation and personalized email auto-reply,
without giving away users' private data. However, on-device LLMs still suffer
from unacceptably long inference latency, especially the time to first token
(prefill stage) due to the need of long context for accurate, personalized
content generation, as well as the lack of parallel computing capacity of
mobile CPU/GPU.
  To enable practical on-device LLM, we present mllm-NPU, the first-of-its-kind
LLM inference system that efficiently leverages on-device Neural Processing
Unit (NPU) offloading. Essentially, mllm-NPU is an algorithm-system co-design
that tackles a few semantic gaps between the LLM architecture and contemporary
NPU design. Specifically, it re-constructs the prompt and model in three
levels: (1) At prompt level, it divides variable-length prompts into multiple
fixed-sized chunks while maintaining data dependencies; (2) At tensor level, it
identifies and extracts significant outliers to run on the CPU/GPU in parallel
with minimal overhead; (3) At block level, it schedules Transformer blocks in
an out-of-order manner to the CPU/GPU and NPU based on their hardware affinity
and sensitivity to accuracy. Compared to competitive baselines, mllm-NPU
achieves 22.4x faster prefill speed and 30.7x energy savings on average, and up
to 32.8x speedup in an end-to-end real-world application. For the first time,
mllm-NPU achieves more than 1,000 tokens/sec prefilling for a billion-sized
model (Qwen1.5-1.8B), paving the way towards practical on-device LLM.
