  Algorithmic transparency entails exposing system properties to various
stakeholders for purposes that include understanding, improving, and contesting
predictions. Until now, most research into algorithmic transparency has
predominantly focused on explainability. Explainability attempts to provide
reasons for a machine learning model's behavior to stakeholders. However,
understanding a model's specific behavior alone might not be enough for
stakeholders to gauge whether the model is wrong or lacks sufficient knowledge
to solve the task at hand. In this paper, we argue for considering a
complementary form of transparency by estimating and communicating the
uncertainty associated with model predictions. First, we discuss methods for
assessing uncertainty. Then, we characterize how uncertainty can be used to
mitigate model unfairness, augment decision-making, and build trustworthy
systems. Finally, we outline methods for displaying uncertainty to stakeholders
and recommend how to collect information required for incorporating uncertainty
into existing ML pipelines. This work constitutes an interdisciplinary review
drawn from literature spanning machine learning, visualization/HCI, design,
decision-making, and fairness. We aim to encourage researchers and
practitioners to measure, communicate, and use uncertainty as a form of
transparency.
