  Recent studies have revealed that, during the inference on generative AI
models such as transformer, the importance of different weights exhibits
substantial context-dependent variations. This naturally manifests a promising
potential of adaptively configuring weight quantization to improve the
generative AI inference efficiency. Although configurable weight quantization
can readily leverage the hardware support of variable-precision arithmetics in
modern GPU and AI accelerators, little prior research has studied how one could
exploit variable weight quantization to proportionally improve the AI model
memory access speed and energy efficiency. Motivated by the rapidly maturing
CXL ecosystem, this work develops a CXL-based design solution to fill this gap.
The key is to allow CXL memory controllers play an active role in supporting
and exploiting runtime configurable weight quantization. Using transformer as a
representative generative AI model, we carried out experiments that well
demonstrate the effectiveness of the proposed design solution.
