  Large Language Models (LLMs) have recently demonstrated remarkable success
across various tasks. However, efficiently serving LLMs has been a challenge
due to the large memory bottleneck, specifically in small batch inference
settings (e.g. mobile devices). Weight-only quantization can be a promising
approach, but sub-4 bit quantization remains a challenge due to large-magnitude
activation outliers. To mitigate the undesirable outlier effect, we first
propose per-IC quantization, a simple yet effective method that creates
quantization groups within each input channel (IC) rather than the conventional
per-output-channel (per-OC). Our method is motivated by the observation that
activation outliers affect the input dimension of the weight matrix, so
similarly grouping the weights in the IC direction can isolate outliers within
a group. We also find that activation outliers do not dictate quantization
difficulty, and inherent weight sensitivities also exist. With per-IC
quantization as a new outlier-friendly scheme, we propose Adaptive Dimensions
(AdaDim), a versatile quantization framework that can adapt to various weight
sensitivity patterns. We demonstrate the effectiveness of AdaDim by augmenting
prior methods such as Round-To-Nearest and GPTQ, showing significant
improvements across various language modeling benchmarks for both base (up to
+4.7% on MMLU) and instruction-tuned (up to +10% on HumanEval) LLMs. Code is
available at https://github.com/johnheo/adadim-llm
