  This work examines the challenges of training neural networks using vector
quantization using straight-through estimation. We find that a primary cause of
training instability is the discrepancy between the model embedding and the
code-vector distribution. We identify the factors that contribute to this
issue, including the codebook gradient sparsity and the asymmetric nature of
the commitment loss, which leads to misaligned code-vector assignments. We
propose to address this issue via affine re-parameterization of the code
vectors. Additionally, we introduce an alternating optimization to reduce the
gradient error introduced by the straight-through estimation. Moreover, we
propose an improvement to the commitment loss to ensure better alignment
between the codebook representation and the model embedding. These optimization
methods improve the mathematical approximation of the straight-through
estimation and, ultimately, the model performance. We demonstrate the
effectiveness of our methods on several common model architectures, such as
AlexNet, ResNet, and ViT, across various tasks, including image classification
and generative modeling.
