  The large language model era urges faster and less costly inference. Prior
model compression works on LLMs tend to undertake a software-centric approach
primarily focused on the simulated quantization performance. By neglecting the
feasibility of deployment, these approaches are typically disabled in real
practice. They used to drastically push down the quantization bit range for a
reduced computation which might not be supported by the mainstream hardware, or
involve sophisticated algorithms that introduce extra computation or memory
access overhead. We argue that pursuing a hardware-centric approach in the
construction of quantization algorithms is crucial. In this regard, we are
driven to build our compression method on top of hardware awareness,
eliminating impractical algorithm choices while maximizing the benefit of
hardware acceleration. Our method, OdysseyLLM, comes with a novel W4A8 kernel
implementation called FastGEMM and a combined recipe of quantization
strategies. Extensive experiments manifest the superiority of our W4A8 method
which brings the actual speed boosting up to \textbf{4$\times$} compared to
Hugging Face FP16 inference and \textbf{2.23$\times$} vs. the state-of-the-art
inference engine TensorRT-LLM in FP16, and \textbf{1.45$\times$} vs.
TensorRT-LLM in INT8, yet without substantially harming the performance.
