  Recently proposed methods for 1-bit and 1.58-bit quantization aware training
investigate the performance and behavior of these methods in the context of
large language models, finding state-of-the-art performance for models with
more than 3B parameters. In this work, we investigate 1.58-bit quantization for
small language and vision models ranging from 100K to 48M parameters. We
introduce a variant of BitNet b1.58, which allows to rely on the median rather
than the mean in the quantization process.
  Through extensive experiments we investigate the performance of 1.58-bit
models obtained through quantization aware training. We further investigate the
robustness of 1.58-bit quantization-aware training to changes in the learning
rate and regularization through weight decay, finding different patterns for
small language and vision models than previously reported for large language
models.
  Our results showcase that 1.58-bit quantization-aware training provides
state-of-the-art performance for small language models when doubling hidden
layer sizes and reaches or even surpasses state-of-the-art performance for
small vision models of identical size. Ultimately, we demonstrate that 1.58-bit
quantization-aware training is a viable and promising approach also for
training smaller deep learning networks, facilitating deployment of such models
in low-resource use-cases and encouraging future research.
