  Efficiently serving neural network models with low latency is becoming more
challenging due to increasing model complexity and parameter count. Model
quantization offers a solution which simultaneously reduces memory footprint
and compute requirements. However, aggressive quantization may lead to an
unacceptable loss in model accuracy owing to differences in sensitivity to
numerical imperfection across different layers in the model. To address this
challenge, we propose a mixed-precision post training quantization (PTQ)
approach that assigns different numerical precisions to tensors in a network
based on their specific needs, for a reduced memory footprint and improved
latency while preserving model accuracy. Previous works rely on layer-wise
Hessian information to determine numerical precision, but as we demonstrate,
Hessian estimation is typically insufficient in determining an effective
ordering of layer sensitivities. We address this by augmenting the estimated
Hessian with additional information to capture inter-layer dependencies. We
demonstrate that this consistently improves PTQ performance along the
accuracy-latency Pareto frontier across multiple models. Our method combines
second-order information and inter-layer dependencies to guide a bisection
search, finding quantization configurations within a user-configurable model
accuracy degradation range. We evaluate the effectiveness of our method on the
ResNet50, MobileNetV2, and BERT models. Our experiments demonstrate latency
reductions compared to a 16-bit baseline of $25.48\%$, $21.69\%$, and $33.28\%$
respectively, while maintaining model accuracy to within $99.99\%$ of the
baseline model.
