  With increasing deployment of machine learning systems in various real-world
tasks, there is a greater need for accurate quantification of predictive
uncertainty. While the common goal in uncertainty quantification (UQ) in
machine learning is to approximate the true distribution of the target data,
many works in UQ tend to be disjoint in the evaluation metrics utilized, and
disparate implementations for each metric lead to numerical results that are
not directly comparable across different works. To address this, we introduce
Uncertainty Toolbox, an open-source python library that helps to assess,
visualize, and improve UQ. Uncertainty Toolbox additionally provides
pedagogical resources, such as a glossary of key terms and an organized
collection of key paper references. We hope that this toolbox is useful for
accelerating and uniting research efforts in uncertainty in machine learning.
