  The notion that algorithmic systems should be "transparent" and "explainable"
is common in the many statements of consensus principles developed by
governments, companies, and advocacy organizations. But what exactly do policy
and legal actors want from these technical concepts, and how do their
desiderata compare with the explainability techniques developed in the machine
learning literature? In hopes of better connecting the policy and technical
communities, we provide case studies illustrating five ways in which
algorithmic transparency and explainability have been used in policy settings:
specific requirements for explanations; in nonbinding guidelines for internal
governance of algorithms; in regulations applicable to highly regulated
settings; in guidelines meant to increase the utility of legal liability for
algorithms; and broad requirements for model and data transparency. The case
studies span a spectrum from precise requirements for specific types of
explanations to nonspecific requirements focused on broader notions of
transparency, illustrating the diverse needs, constraints, and capacities of
various policy actors and contexts. Drawing on these case studies, we discuss
promising ways in which transparency and explanation could be used in policy,
as well as common factors limiting policymakers' use of algorithmic
explainability. We conclude with recommendations for researchers and
policymakers.
